Oct 30 20:16:49.312364 ip-10-0-141-29 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 20:16:49.312377 ip-10-0-141-29 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 20:16:49.312387 ip-10-0-141-29 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 20:16:49.312889 ip-10-0-141-29 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 20:16:59.497548 ip-10-0-141-29 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 20:16:59.497565 ip-10-0-141-29 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 39919e087b864812829e49232f14df14 --
Oct 30 20:18:50.913967 ip-10-0-141-29 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:18:51.341223 ip-10-0-141-29 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:18:51.341223 ip-10-0-141-29 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:18:51.341223 ip-10-0-141-29 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:18:51.341223 ip-10-0-141-29 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:18:51.345830 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345718    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.345830 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345824    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.345830 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345829    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.345830 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345834    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.345830 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345837    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.345830 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345841    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345845    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345849    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345853    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345857    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345860    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345864    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345868    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345872    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345876    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345880    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345884    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345888    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345892    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345896    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345899    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345902    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345906    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345910    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345913    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.346196 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345917    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345921    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345924    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345931    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345934    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345938    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345942    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345945    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345950    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345954    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345957    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345962    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345966    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345970    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345974    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345978    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345982    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345987    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345991    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345995    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.347010 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.345999    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346004    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346009    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346013    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346018    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346022    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346026    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346030    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346034    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346038    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346042    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346048    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346052    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346056    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346061    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346066    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346070    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346074    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346078    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.347847 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346082    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346087    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346091    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346095    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346100    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346104    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346108    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346112    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346118    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346122    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346126    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346130    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346135    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346139    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346143    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346147    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346151    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346155    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346160    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.348645 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346166    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346170    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346174    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346179    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346183    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346188    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346192    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346200    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346204    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346208    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346213    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346220    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346229    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346235    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346239    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346244    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346252    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346258    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.349258 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346262    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346267    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346272    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.346277    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347702    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347714    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347718    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347724    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347728    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347733    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347737    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347742    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347746    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347750    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347754    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347759    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347763    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347767    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347772    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347776    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.349756 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347781    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347785    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347789    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347793    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347797    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347801    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347805    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347810    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347815    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347819    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347823    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347828    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347832    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347836    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347841    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347844    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347850    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347854    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347858    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347862    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.350244 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347866    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347870    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347874    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347878    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347882    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347887    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347891    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347895    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347899    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347903    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347908    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347912    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347916    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347921    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347925    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347929    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347933    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347937    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347942    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347946    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.350771 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347950    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347957    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347963    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347967    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347971    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347976    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347980    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347984    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347989    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347994    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.347998    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348005    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348012    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348017    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348022    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348026    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348032    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348036    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348041    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.351245 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348045    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348049    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348053    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348059    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348064    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348068    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348072    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348076    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348080    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348085    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348089    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348094    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348098    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348102    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348107    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348111    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348116    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348119    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348124    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.351717 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348128    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348133    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348137    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348141    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348146    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348150    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348154    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348158    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348162    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348167    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348171    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.348175    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349134    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349148    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349151    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349154    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349157    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349160    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349163    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349166    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.352266 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349169    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349172    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349174    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349177    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349180    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349183    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349185    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349188    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349190    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349193    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349195    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349198    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349200    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349203    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349206    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349210    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349212    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349217    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349221    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.352788 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349224    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349227    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349230    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349233    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349236    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349239    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349243    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349245    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349248    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349251    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349253    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349256    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349259    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349262    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349265    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349268    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349270    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349281    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349285    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349288    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.353243 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349291    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349294    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349296    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349300    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349303    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349305    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349308    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349311    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349314    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349316    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349319    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349321    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349324    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349326    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349329    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349331    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349334    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349336    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349339    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.353770 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349343    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349346    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349349    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349352    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349354    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349357    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349359    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349362    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349365    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349368    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349371    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349373    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349376    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349378    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349381    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349383    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349386    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349389    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349391    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349394    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.354263 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349397    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349399    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349402    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349405    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349407    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349410    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349412    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349414    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349417    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349420    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349422    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349425    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349428    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349431    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349435    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349438    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349440    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349442    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349445    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.349448    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.354759 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350471    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350484    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350493    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350499    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350504    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350507    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350512    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350516    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350520    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350523    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350527    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350548    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350551    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350555    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350558    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350561    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350564    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350567    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350572    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350575    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350578    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350582    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350585    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350591    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:18:51.355207 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350594    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350598    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350602    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350605    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350608    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350611    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350615    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350617    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350622    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350625    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350628    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350631    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350635    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350637    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350644    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350647    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350650    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350654    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350657    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350662    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350665    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350668    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350671    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350674    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350677    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:18:51.355784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350680    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350683    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350686    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350689    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350692    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350696    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350700    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350703    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350707    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350710    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350714    2576 flags.go:64] FLAG: --help="false"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350718    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350721    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350724    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350727    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350731    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350734    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350737    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350740    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350744    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350746    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350749    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350753    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350756    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:18:51.356434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350758    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350761    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350765    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350769    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350771    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350774    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350777    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350780    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350787    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350790    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350792    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350795    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350798    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350802    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350805    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350808    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350813    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350816    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350820    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350824    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350827    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350830    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350833    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350836    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350839    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:18:51.357037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350842    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350851    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350855    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350857    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350861    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350864    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350866    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350869    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350872    2576 flags.go:64] FLAG: --port="10250"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350876    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350878    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-07b37be0b914f8226"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350882    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350885    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350888    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350891    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350895    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350898    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350901    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350904    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350908    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350912    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350915    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350918    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350921    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350924    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350928    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350931    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:18:51.357693 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350935    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350938    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350941    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350944    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350948    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350951    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350953    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350956    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350959    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350962    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350969    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350972    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350978    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350982    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350984    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350989    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350992    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350995    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.350998    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.351001    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.351004    2576 flags.go:64] FLAG: --v="2"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.351009    2576 flags.go:64] FLAG: --version="false"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.351014    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.351018    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.351021    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:18:51.358345 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351194    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351199    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351202    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351205    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351208    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351210    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351213    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351216    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351219    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351222    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351224    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351227    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351229    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351232    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351234    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351237    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351240    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351244    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351246    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.358976 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351249    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351252    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351254    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351257    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351260    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351262    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351265    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351268    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351270    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351273    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351275    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351278    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351280    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351283    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351285    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351288    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351290    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351292    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351295    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.359432 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351299    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351304    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351310    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351314    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351317    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351320    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351322    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351326    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351329    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351332    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351335    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351339    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351341    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351343    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351346    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351348    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351351    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351353    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351356    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.359907 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351358    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351361    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351364    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351366    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351369    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351371    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351374    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351377    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351379    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351381    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351384    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351386    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351389    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351392    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351395    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351398    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351401    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351404    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351407    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351410    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.360366 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351412    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351415    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351417    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351419    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351423    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351426    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351428    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351432    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351434    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351437    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351439    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351441    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351444    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351447    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351449    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351452    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351455    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351457    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351460    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.360890 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351462    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351465    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351468    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351470    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351473    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351476    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351478    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351481    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351484    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351487    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.351516    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351570    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351575    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351579    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351583    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351586    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.361350 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351589    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351596    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351599    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351602    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351605    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351607    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351610    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351613    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351616    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351619    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351621    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351624    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351626    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351629    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351632    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351635    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351638    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351640    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351643    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.361752 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351645    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351648    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351651    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351653    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351656    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351658    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351663    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351666    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351668    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351672    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351674    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351677    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351680    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351682    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351686    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351689    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351692    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351694    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351697    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351699    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.362195 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351702    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351704    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351707    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351709    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351712    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351715    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351717    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351720    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351722    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351725    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351728    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351731    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351733    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351736    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351740    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351744    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351747    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351750    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351755    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.362685 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351759    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351762    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351765    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351770    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351774    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351776    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351779    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351783    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351785    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351788    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351790    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351792    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351795    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351797    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351800    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351802    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351805    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351807    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351810    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351812    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.363157 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351815    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351818    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351820    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351822    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351825    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351827    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351831    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351833    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351836    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351838    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351841    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351844    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351847    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351849    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351852    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351854    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351858    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351860    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351863    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351867    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.363650 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351869    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351872    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.351874    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.361518    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.361552    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.361570    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.361575    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361650    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361656    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361660    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361663    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361666    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361669    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361671    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361674    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361677    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361680    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361683    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361686    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361689    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.364113 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361691    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361696    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361700    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361703    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361706    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361709    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361712    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361714    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361723    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361726    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361728    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361731    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361733    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361736    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361738    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361741    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361744    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361747    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361749    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.364769 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361752    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361754    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361757    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361760    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361762    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361765    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361767    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361770    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361773    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361776    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361778    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361780    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361783    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361785    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361788    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361790    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361792    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361795    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361797    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361800    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.365216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361802    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361804    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361813    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361816    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361818    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361821    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361823    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361826    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361829    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361832    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361835    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361837    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361840    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361842    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361845    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361847    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361850    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361852    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361854    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.365724 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361858    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361862    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361865    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361868    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361870    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361873    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361875    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361878    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361880    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361882    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361885    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361888    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361890    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361900    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361903    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361905    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361913    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361916    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361918    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.366179 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361921    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361924    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361926    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361930    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361932    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361935    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361937    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361940    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361943    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361945    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361948    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361950    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361953    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361955    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361958    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.361961    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.366634 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.361990    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362182    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362187    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362190    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362192    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362195    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362198    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362200    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362203    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362206    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362208    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362211    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362213    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362215    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362223    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362226    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362229    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362231    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362235    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362239    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.367004 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362242    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362245    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362248    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362250    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362254    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362257    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362260    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362263    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362265    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362268    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362270    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362273    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362275    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362278    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362280    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362282    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362285    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362287    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362290    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362292    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.367489 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362295    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362297    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362300    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362302    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362305    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362307    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362309    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362319    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362321    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362324    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362327    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362329    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362332    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362334    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362337    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362339    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362342    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362344    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362346    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362349    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.367965 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362352    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362354    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362356    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362359    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362361    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362364    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362366    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362368    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362371    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362373    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362375    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362378    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362380    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362383    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362385    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362387    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362390    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362392    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362394    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.368430 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362396    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362405    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362408    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362411    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362413    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362416    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362419    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362421    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362424    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362426    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362429    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362431    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362433    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362436    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362439    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362441    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362444    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362446    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362449    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362451    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.368932 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362453    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362456    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362459    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362461    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362463    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362466    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362468    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362471    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.362494    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362546    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362551    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362554    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362556    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362559    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362567    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362570    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:18:51.369407 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362573    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362582    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362585    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362587    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362590    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362592    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362594    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362597    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362599    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362602    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362604    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362607    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362609    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362611    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362614    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362616    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362619    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362621    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362624    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362627    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:18:51.370027 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362629    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362632    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362634    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362637    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362639    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362642    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362644    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362647    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362649    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362652    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362654    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362662    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362665    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362667    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362669    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362672    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362674    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362677    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362679    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362681    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:18:51.370816 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362684    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362688    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362692    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362696    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362699    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362702    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362705    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362709    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362712    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362715    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362717    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362720    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362722    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362725    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362727    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362730    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362732    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362735    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362737    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:18:51.371390 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362740    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362742    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362744    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362747    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362749    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362762    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362764    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362767    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362769    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362772    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362774    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362777    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362780    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362782    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362785    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362787    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362789    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362792    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362794    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362796    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:18:51.371917 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362799    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362801    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362804    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362806    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362809    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362811    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362813    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362816    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362818    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362826    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362829    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362831    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362833    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362836    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362838    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362841    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362847    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362849    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362857    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:18:51.372389 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:51.362860    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:18:51.372943 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.362918    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:18:51.372943 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.365338    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:18:51.372943 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.366267    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 20:18:51.372943 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.366374    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:18:51.372943 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.366415    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:18:51.372943 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.370731    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:18:51.375661 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.375644    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:18:51.386210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.386194    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:18:51.389866 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.389850    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 20:18:51.390726 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.390699    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:18:51.394995 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.394970    2576 fs.go:135] Filesystem UUIDs: map[2aaa55d8-3651-47e0-be96-149c87065615:/dev/nvme0n1p4 41468992-4b9f-4a5e-8c12-4cd808a87e30:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 20:18:51.395080 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.394995    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:18:51.400393 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.400258    2576 manager.go:217] Machine: {Timestamp:2026-10-30 20:18:51.398839136 +0000 UTC m=+0.357085387 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078458 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2813f99d7b3267d6368a7c0208cbbb SystemUUID:ec2813f9-9d7b-3267-d636-8a7c0208cbbb BootID:39919e08-7b86-4812-829e-49232f14df14 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:bb:60:53:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:bb:60:53:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:1d:9c:99:8e:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:18:51.400961 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.400946    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:18:51.401087 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.401069    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:18:51.404586 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.404553    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:18:51.404734 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.404588    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-29.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:18:51.404783 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.404736    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:18:51.404783 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.404744    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:18:51.404783 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.404771    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:18:51.404859 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.404785    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:18:51.405921 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.405909    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:18:51.410219 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.410204    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:18:51.410355 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.410345    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:18:51.411246 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.411234    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:18:51.411280 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.411249    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:18:51.411280 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.411261    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:18:51.411280 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.411273    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:18:51.411394 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.411282    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:18:51.411757 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.411742    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:18:51.414112 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.414089    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:18:51.419622 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.419599    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:18:51.420596 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.420580    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:18:51.420669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.420613    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:18:51.421440 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.421429    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:18:51.421474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.421446    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:18:51.421474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.421452    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:18:51.421474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.421459    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:18:51.421474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.421464    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:18:51.421474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.421470    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:18:51.422102 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.422093    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:18:51.422102 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.422102    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:18:51.422158 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.422111    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:18:51.422158 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.422117    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:18:51.422158 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.422127    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:18:51.422158 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.422136    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:18:51.423625 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.423615    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:18:51.423625 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.423625    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:18:51.429008 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.428711    2576 server.go:1263] "Started kubelet"
Oct 30 20:18:51.429118 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.429057    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:18:51.429247 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.429128    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:18:51.429336 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.429136    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:18:51.430002 ip-10-0-141-29 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:18:51.431031 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.431015    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:18:51.432345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.432329    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:18:51.436482 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.436458    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:18:51.441316 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.441294    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:18:51.442574 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.442553    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:18:51.442753 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.442725    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:51.443447 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.443430    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:18:51.443570 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.443449    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:18:51.443570 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.443430    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:18:51.443659 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.443589    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:18:51.443659 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.443597    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:18:51.445142 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.445124    2576 factory.go:55] Registering systemd factory
Oct 30 20:18:51.445142 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.445143    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:18:51.445421 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.445403    2576 factory.go:153] Registering CRI-O factory
Oct 30 20:18:51.445462 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.445428    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:18:51.445515 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.445503    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:18:51.445578 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.445553    2576 factory.go:103] Registering Raw factory
Oct 30 20:18:51.445578 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.445573    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:18:51.447187 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.447172    2576 manager.go:319] Starting recovery of all containers
Oct 30 20:18:51.457491 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.457469    2576 manager.go:324] Recovery completed
Oct 30 20:18:51.462105 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.462086    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.463808 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.463777    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.463890 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.463837    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.463890 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.463848    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.464463 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.464448    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:18:51.464463 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.464459    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:18:51.464616 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.464475    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.464616 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.464476    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:18:51.466790 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.466775    2576 policy_none.go:50] "Start"
Oct 30 20:18:51.466866 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.466795    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:18:51.466866 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.466808    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:18:51.468869 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.468855    2576 policy_none.go:44] "Start"
Oct 30 20:18:51.471749 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.471714    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.471855 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.471761    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.471855 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.471783    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.524501 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.524275    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:18:51.529368 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.529349    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:18:51.529839 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.529821    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:18:51.529901 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.529842    2576 server.go:87] "Starting device plugin registration server"
Oct 30 20:18:51.530128 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.530114    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:18:51.530196 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.530129    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:18:51.530239 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.530221    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:18:51.530318 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.530304    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:18:51.530318 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.530318    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:18:51.530951 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.530926    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:18:51.531032 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.530984    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:51.568408 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.568377    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-29.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:18:51.568408 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.568389    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-29.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:18:51.570369 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.568369    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.us-west-1.compute.internal.18e3681895ffd388  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:18:51.4286642 +0000 UTC m=+0.386910454,LastTimestamp:2026-10-30 20:18:51.4286642 +0000 UTC m=+0.386910454,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.us-west-1.compute.internal,}"
Oct 30 20:18:51.593689 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.593609    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:18:51.595004 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.594984    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:18:51.595061 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.595017    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:18:51.595061 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.595043    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:18:51.595139 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.595087    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 20:18:51.628307 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.628173    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.us-west-1.compute.internal.18e3681898180dc2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-29.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:18:51.463806402 +0000 UTC m=+0.422052653,LastTimestamp:2026-10-30 20:18:51.463806402 +0000 UTC m=+0.422052653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.us-west-1.compute.internal,}"
Oct 30 20:18:51.628510 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.628488    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 20:18:51.630643 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.630628    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.631451 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.631435    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.631525 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.631463    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.631525 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.631473    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.631525 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.631502    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.647783 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.647753    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:18:51.692124 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.692094    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c26dd"
Oct 30 20:18:51.695401 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.695382    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"]
Oct 30 20:18:51.695486 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.695450    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.696387 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.696371    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.696456 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.696399    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.696456 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.696420    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.698649 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.698637    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.698845 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.698827    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.698888 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.698873    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.699548 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.699514    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.699646 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.699556    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.699646 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.699568    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.699646 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.699590    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.699646 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.699611    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.699646 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.699623    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.699860 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.699844    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c26dd"
Oct 30 20:18:51.701575 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.701559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.701652 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.701590    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.702283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.702267    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.702365 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.702297    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.702365 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.702311    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.726690 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.726643    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-29.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 20:18:51.726690 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.726642    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 20:18:51.726934 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.726743    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.us-west-1.compute.internal.18e3681898189a67  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-29.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:18:51.463842407 +0000 UTC m=+0.422088657,LastTimestamp:2026-10-30 20:18:51.463842407 +0000 UTC m=+0.422088657,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.us-west-1.compute.internal,}"
Oct 30 20:18:51.738695 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.738659    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.741710 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.741686    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.744925 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.744900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.745009 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.744938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.745009 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.744957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a97eddfe748e1e815e76819b9d90360d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"a97eddfe748e1e815e76819b9d90360d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.784490 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.784460    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:18:51.785028 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.785009    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-29.us-west-1.compute.internal\" not found" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.845848 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.845761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.845848 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.845798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a97eddfe748e1e815e76819b9d90360d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"a97eddfe748e1e815e76819b9d90360d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.845848 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.845824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.846075 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.845879    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.846075 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.845882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.846075 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.845882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a97eddfe748e1e815e76819b9d90360d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"a97eddfe748e1e815e76819b9d90360d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.870902 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:51.870871    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-29.us-west-1.compute.internal\" not found" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:51.986020 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.985959    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:51.987117 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.987098    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:51.987238 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.987131    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:51.987238 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.987160    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:51.987238 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:51.987189    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:52.040798 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.040751    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:52.043949 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.043927    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:52.044048 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.043954    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-29.us-west-1.compute.internal\": node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.044099 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.044085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:52.168523 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.168430    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.268615 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.268575    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.366828 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.366788    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:18:52.367375 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.366969    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:18:52.367375 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.367049    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab4a4e8a057d64e13b2573a04657ce20-dcfeed6681acbb51.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-29.us-west-1.compute.internal.18e3681898180dc2\": read tcp 10.0.141.29:56686->77.112.75.128:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-29.us-west-1.compute.internal.18e3681898180dc2  default   4751 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-29.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:18:51 +0000 UTC,LastTimestamp:2026-10-30 20:18:51.696385139 +0000 UTC m=+0.654631390,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.us-west-1.compute.internal,}"
Oct 30 20:18:52.368934 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.368916    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.469737 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.469655    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.569782 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.569745    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.592765 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.592737    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:18:52.670753 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.670721    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.704121 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.704050    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:13:52 +0000 UTC" deadline="2028-08-11 10:53:57.589534298 +0000 UTC"
Oct 30 20:18:52.704121 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.704115    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15614h35m4.885423117s"
Oct 30 20:18:52.715214 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.715197    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:18:52.716200 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.716179    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:18:52.716320 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.716217    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:18:52.716320 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.716231    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:18:52.770952 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:52.770922    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:18:52.830330 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.830291    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:18:52.842883 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.842854    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:52.909761 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.909727    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:18:52.911572 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.911555    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:18:52.974407 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:52.974369    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:18:53.255993 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.255944    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97eddfe748e1e815e76819b9d90360d.slice/crio-1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac WatchSource:0}: Error finding container 1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac: Status 404 returned error can't find the container with id 1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac
Oct 30 20:18:53.256216 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.256195    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2203f4a44f682afa473e5863b4adcd60.slice/crio-9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11 WatchSource:0}: Error finding container 9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11: Status 404 returned error can't find the container with id 9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11
Oct 30 20:18:53.260567 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.260549    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:18:53.413164 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.413078    2576 apiserver.go:52] "Watching apiserver"
Oct 30 20:18:53.428202 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.428167    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:18:53.428332 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.428274    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:18:53.443072 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.443040    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:18:53.523460 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.523421    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:18:53.523893 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.523868    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58","openshift-cluster-node-tuning-operator/tuned-zz2fw","openshift-multus/multus-additional-cni-plugins-r25dm","openshift-multus/multus-g6df8","openshift-ovn-kubernetes/ovnkube-node-6vhjk","openshift-dns/node-resolver-b5kfh","openshift-image-registry/node-ca-x2lgj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-svcz9","openshift-network-operator/iptables-alerter-nv9fr","kube-system/global-pull-secret-syncer-jkspb","kube-system/konnectivity-agent-l4ldp","kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-n6f9w"]
Oct 30 20:18:53.527756 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.527733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.529844 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.529823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.531993 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.531971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.534078 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.534057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.536368 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.536349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.538499 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.538481    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.540898 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.540882    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.544411 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.544380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:53.544567 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.544514    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:18:53.548285 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.548266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.550781 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.550761    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.550889 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.550833    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:18:53.553150 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.553127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.555360 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-cnibin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.555434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-netns\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.555434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-multus\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.555434 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-log-socket\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.555689 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.555689 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:53.555689 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.555616    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:18:53.555689 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-kubelet\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.555689 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-script-lib\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-system-cni-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555769    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-var-lib-kubelet\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-cni-binary-copy\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-systemd-units\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-ovn\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-node-log\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-env-overrides\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.555955 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-sys-fs\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.555981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksnbn\" (UniqueName: \"kubernetes.io/projected/62ad0a1a-eede-46f0-8e64-88b9d86a5239-kube-api-access-ksnbn\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfmw\" (UniqueName: \"kubernetes.io/projected/f6244e4c-72f8-40ce-9a76-2afc5467cebe-kube-api-access-hvfmw\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-slash\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-systemd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-conf\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-registration-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-lib-modules\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/2b0926a1-74b6-4510-8fbc-91609d959cbf-kube-api-access-srm2k\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.556357 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6244e4c-72f8-40ce-9a76-2afc5467cebe-tmp-dir\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-host\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-system-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-conf-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-device-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysconfig\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-config\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfdd\" (UniqueName: \"kubernetes.io/projected/2d15d48f-5197-4096-9011-d82bcb96fde9-kube-api-access-8xfdd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cnibin\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-kubernetes\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-socket-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-k8s-cni-cncf-io\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.556885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hxx\" (UniqueName: \"kubernetes.io/projected/3084f525-41f6-44e7-9cf4-e0a85e23362e-kube-api-access-k2hxx\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-systemd\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-tuned\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.556977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-bin\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-binary-copy\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6244e4c-72f8-40ce-9a76-2afc5467cebe-hosts-file\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-run\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-multus-certs\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-var-lib-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557146    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-bin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-kubelet\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpl42\" (UniqueName: \"kubernetes.io/projected/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kube-api-access-kpl42\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.557409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-sys\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rwlr\" (UniqueName: \"kubernetes.io/projected/7de73755-c298-441d-a185-bdaf04d2f7ea-kube-api-access-5rwlr\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-etc-kubernetes\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-netns\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-etc-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/877333f1-3281-4c9a-84be-ac1de10369e4-serviceca\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-socket-dir-parent\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-hostroot\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-daemon-config\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d15d48f-5197-4096-9011-d82bcb96fde9-ovn-node-metrics-cert\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-modprobe-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-tmp\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-netd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/877333f1-3281-4c9a-84be-ac1de10369e4-host\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.557948 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.557755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbjk\" (UniqueName: \"kubernetes.io/projected/877333f1-3281-4c9a-84be-ac1de10369e4-kube-api-access-dxbjk\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.584355 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.584329    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.584494 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.584353    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.584737 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.584716    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:18:53.585123 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.585102    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lm5vf\""
Oct 30 20:18:53.585272 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.585206    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwff\""
Oct 30 20:18:53.585826 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.585795    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.585904 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.585886    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.587461 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.587435    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:18:53.587461 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.587449    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.587616 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.587492    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:18:53.587616 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.587526    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:18:53.587724 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.587656    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwdx\""
Oct 30 20:18:53.587770 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.587731    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.589232 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.589217    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:18:53.589407 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.589384    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k74qc\""
Oct 30 20:18:53.592056 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.592034    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.592142 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.592043    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.592142 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.592070    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:18:53.592142 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.592105    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:18:53.599667 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.599628    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"a97eddfe748e1e815e76819b9d90360d","Type":"ContainerStarted","Data":"1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac"}
Oct 30 20:18:53.600659 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.600615    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"2203f4a44f682afa473e5863b4adcd60","Type":"ContainerStarted","Data":"9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11"}
Oct 30 20:18:53.632654 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.632623    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:18:53.632924 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.632906    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5x4ch\""
Oct 30 20:18:53.632994 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.632952    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:18:53.633084 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.633066    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.633730 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.633377    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6wxmv\""
Oct 30 20:18:53.635780 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.635646    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gbmn9\""
Oct 30 20:18:53.637488 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637471    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vn2fz\""
Oct 30 20:18:53.637589 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637576    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.637655 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637619    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:18:53.637712 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637686    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.637712 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637702    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637720    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637721    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637750    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637759    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637755    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637761    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637784    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rxckq"
Oct 30 20:18:53.637822 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.637792    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kv88f\""
Oct 30 20:18:53.645235 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.645218    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:18:53.658939 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.658912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-binary-copy\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.659008 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.658942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6244e4c-72f8-40ce-9a76-2afc5467cebe-hosts-file\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.659008 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.658963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-run\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.659008 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.658980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-multus-certs\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659008 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-var-lib-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659042    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6244e4c-72f8-40ce-9a76-2afc5467cebe-hosts-file\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-multus-certs\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-var-lib-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-bin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659139    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-run\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.659184 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659182    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-bin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659227    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659254    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-kubelet\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpl42\" (UniqueName: \"kubernetes.io/projected/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kube-api-access-kpl42\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.659339    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-kubelet\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659385    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-sys\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-sys\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.659461    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:18:54.159386865 +0000 UTC m=+3.117633123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwlr\" (UniqueName: \"kubernetes.io/projected/7de73755-c298-441d-a185-bdaf04d2f7ea-kube-api-access-5rwlr\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-etc-kubernetes\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-netns\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-etc-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659614    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-etc-kubernetes\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.659669 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-etc-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-netns\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659743    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659749    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-binary-copy\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/877333f1-3281-4c9a-84be-ac1de10369e4-serviceca\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-socket-dir-parent\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-hostroot\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659855    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-daemon-config\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659922    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-hostroot\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-socket-dir-parent\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.659995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d15d48f-5197-4096-9011-d82bcb96fde9-ovn-node-metrics-cert\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-iptables-alerter-script\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-modprobe-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660072    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.660331 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-tmp\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-netd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/877333f1-3281-4c9a-84be-ac1de10369e4-host\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbjk\" (UniqueName: \"kubernetes.io/projected/877333f1-3281-4c9a-84be-ac1de10369e4-kube-api-access-dxbjk\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-modprobe-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-cnibin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-netns\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660325    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/877333f1-3281-4c9a-84be-ac1de10369e4-host\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-multus\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-log-socket\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660375    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/877333f1-3281-4c9a-84be-ac1de10369e4-serviceca\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-cnibin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-netns\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660394    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-kubelet\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-daemon-config\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660459    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-log-socket\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661210 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-script-lib\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-kubelet\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-multus\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-host-slash\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660585    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-netd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtl7\" (UniqueName: \"kubernetes.io/projected/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-kube-api-access-mvtl7\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-system-cni-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-var-lib-kubelet\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-cni-binary-copy\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-systemd-units\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-system-cni-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-ovn\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660823    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-systemd-units\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660887    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-node-log\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-var-lib-kubelet\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.661917 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660949    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-ovn\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-node-log\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.660989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-env-overrides\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-sys-fs\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnbn\" (UniqueName: \"kubernetes.io/projected/62ad0a1a-eede-46f0-8e64-88b9d86a5239-kube-api-access-ksnbn\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-script-lib\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfmw\" (UniqueName: \"kubernetes.io/projected/f6244e4c-72f8-40ce-9a76-2afc5467cebe-kube-api-access-hvfmw\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661089    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-sys-fs\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-slash\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-systemd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-conf\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-registration-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-lib-modules\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/2b0926a1-74b6-4510-8fbc-91609d959cbf-kube-api-access-srm2k\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-dbus\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.662562 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-cni-binary-copy\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661354    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-conf\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-slash\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661372    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-env-overrides\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-registration-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6244e4c-72f8-40ce-9a76-2afc5467cebe-tmp-dir\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661409    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-systemd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-host\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-system-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-conf-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-lib-modules\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-kubelet-config\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661510    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-konnectivity-ca\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661571    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-system-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.663085 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661573    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-host\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661642    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-conf-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-device-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661689    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661724    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-device-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysconfig\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-config\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfdd\" (UniqueName: \"kubernetes.io/projected/2d15d48f-5197-4096-9011-d82bcb96fde9-kube-api-access-8xfdd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-agent-certs\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661812    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysconfig\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661878    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6244e4c-72f8-40ce-9a76-2afc5467cebe-tmp-dir\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cnibin\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-kubernetes\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.661987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-socket-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-k8s-cni-cncf-io\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662039    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-kubernetes\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.663636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-k8s-cni-cncf-io\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662073    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cnibin\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hxx\" (UniqueName: \"kubernetes.io/projected/3084f525-41f6-44e7-9cf4-e0a85e23362e-kube-api-access-k2hxx\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-socket-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-systemd\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-systemd\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662212    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-config\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-tuned\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-bin\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.664186 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.662346    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-bin\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.664624 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.664590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-tuned\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.664754 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.664736    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-tmp\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.664897 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.664878    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d15d48f-5197-4096-9011-d82bcb96fde9-ovn-node-metrics-cert\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.721949 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.721911    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbjk\" (UniqueName: \"kubernetes.io/projected/877333f1-3281-4c9a-84be-ac1de10369e4-kube-api-access-dxbjk\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.722121 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.722073    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwlr\" (UniqueName: \"kubernetes.io/projected/7de73755-c298-441d-a185-bdaf04d2f7ea-kube-api-access-5rwlr\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.722121 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.722071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpl42\" (UniqueName: \"kubernetes.io/projected/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kube-api-access-kpl42\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.723668 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.723634    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/2b0926a1-74b6-4510-8fbc-91609d959cbf-kube-api-access-srm2k\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.723810 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.723785    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfdd\" (UniqueName: \"kubernetes.io/projected/2d15d48f-5197-4096-9011-d82bcb96fde9-kube-api-access-8xfdd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.723810 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.723802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hxx\" (UniqueName: \"kubernetes.io/projected/3084f525-41f6-44e7-9cf4-e0a85e23362e-kube-api-access-k2hxx\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:53.723991 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.723975    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnbn\" (UniqueName: \"kubernetes.io/projected/62ad0a1a-eede-46f0-8e64-88b9d86a5239-kube-api-access-ksnbn\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.724107 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.724093    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfmw\" (UniqueName: \"kubernetes.io/projected/f6244e4c-72f8-40ce-9a76-2afc5467cebe-kube-api-access-hvfmw\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.763549 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.763720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-iptables-alerter-script\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.763720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-host-slash\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.763720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtl7\" (UniqueName: \"kubernetes.io/projected/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-kube-api-access-mvtl7\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.763720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-dbus\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.763720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-kubelet-config\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.763720 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.763674    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:53.763720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-konnectivity-ca\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.764027 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-agent-certs\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.764027 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763724    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-host-slash\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.764027 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.763735    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret podName:96f587e0-8984-434f-b1a5-bae1b56c343b nodeName:}" failed. No retries permitted until 2026-10-30 20:18:54.263719349 +0000 UTC m=+3.221965589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret") pod "global-pull-secret-syncer-jkspb" (UID: "96f587e0-8984-434f-b1a5-bae1b56c343b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:53.764027 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-dbus\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.764027 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763817    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-kubelet-config\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:53.764027 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.763808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:53.764371 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.764349    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-konnectivity-ca\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.764413 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.764349    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-iptables-alerter-script\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.766264 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.766248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-agent-certs\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.822486 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.822449    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:18:53.822486 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.822484    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:18:53.822486 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.822498    2576 projected.go:196] Error preparing data for projected volume kube-api-access-tzwqg for pod openshift-network-diagnostics/network-check-target-n6f9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:53.822763 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:53.822596    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg podName:06d9b61a-283a-4ff0-b132-9319897c7e9b nodeName:}" failed. No retries permitted until 2026-10-30 20:18:54.322572832 +0000 UTC m=+3.280819087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tzwqg" (UniqueName: "kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg") pod "network-check-target-n6f9w" (UID: "06d9b61a-283a-4ff0-b132-9319897c7e9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:53.824920 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.824900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtl7\" (UniqueName: \"kubernetes.io/projected/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-kube-api-access-mvtl7\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.846935 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.846898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:18:53.850078 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.850058    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:18:53.861979 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.861950    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93cda06_ee44_4b46_8a1b_1cee671fa4e7.slice/crio-3de02f77db8a82d890509743cb0efee95efbd63a22919932eedb4237664ef034 WatchSource:0}: Error finding container 3de02f77db8a82d890509743cb0efee95efbd63a22919932eedb4237664ef034: Status 404 returned error can't find the container with id 3de02f77db8a82d890509743cb0efee95efbd63a22919932eedb4237664ef034
Oct 30 20:18:53.862586 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.862563    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de73755_c298_441d_a185_bdaf04d2f7ea.slice/crio-beb2e1d8a32270e38bf7eba4a548f05a7072d7706406ec8a6ba2da857df41405 WatchSource:0}: Error finding container beb2e1d8a32270e38bf7eba4a548f05a7072d7706406ec8a6ba2da857df41405: Status 404 returned error can't find the container with id beb2e1d8a32270e38bf7eba4a548f05a7072d7706406ec8a6ba2da857df41405
Oct 30 20:18:53.866174 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.866153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g6df8"
Oct 30 20:18:53.872323 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.872303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:18:53.875701 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.875675    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0926a1_74b6_4510_8fbc_91609d959cbf.slice/crio-c941c1c439823ba7453351ffba09a6ef0a3f801cfec417e7f7f5bdc7a639de3e WatchSource:0}: Error finding container c941c1c439823ba7453351ffba09a6ef0a3f801cfec417e7f7f5bdc7a639de3e: Status 404 returned error can't find the container with id c941c1c439823ba7453351ffba09a6ef0a3f801cfec417e7f7f5bdc7a639de3e
Oct 30 20:18:53.879261 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.879243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:18:53.880477 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.880459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:18:53.880832 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.880813    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ad0a1a_eede_46f0_8e64_88b9d86a5239.slice/crio-edfcc54666535d456059b2c9f25f78e0a3ab73ae0461ba5d9edd0809589b7c08 WatchSource:0}: Error finding container edfcc54666535d456059b2c9f25f78e0a3ab73ae0461ba5d9edd0809589b7c08: Status 404 returned error can't find the container with id edfcc54666535d456059b2c9f25f78e0a3ab73ae0461ba5d9edd0809589b7c08
Oct 30 20:18:53.889360 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.889324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:18:53.889441 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.889395    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6244e4c_72f8_40ce_9a76_2afc5467cebe.slice/crio-926623d0c661fe33616eaeca447149441d5d295dc2dd90534c80533adb588b21 WatchSource:0}: Error finding container 926623d0c661fe33616eaeca447149441d5d295dc2dd90534c80533adb588b21: Status 404 returned error can't find the container with id 926623d0c661fe33616eaeca447149441d5d295dc2dd90534c80533adb588b21
Oct 30 20:18:53.891422 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.891402    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877333f1_3281_4c9a_84be_ac1de10369e4.slice/crio-b9fc7f744d19c4b75767b7635c4bfdb672ebd771bdfb2eaedce7d3f5d30151e7 WatchSource:0}: Error finding container b9fc7f744d19c4b75767b7635c4bfdb672ebd771bdfb2eaedce7d3f5d30151e7: Status 404 returned error can't find the container with id b9fc7f744d19c4b75767b7635c4bfdb672ebd771bdfb2eaedce7d3f5d30151e7
Oct 30 20:18:53.891506 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.891493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:53.896633 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:53.896604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:18:53.899431 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.899399    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d003e73_1f2d_4e65_a6dc_f8ead9bc6109.slice/crio-beb5e664fd0e98dc0a7e5256f71f5c9df12431436100b8e0f3cc165d097264ec WatchSource:0}: Error finding container beb5e664fd0e98dc0a7e5256f71f5c9df12431436100b8e0f3cc165d097264ec: Status 404 returned error can't find the container with id beb5e664fd0e98dc0a7e5256f71f5c9df12431436100b8e0f3cc165d097264ec
Oct 30 20:18:53.902712 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.902690    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23bf1a3_ba4f_40cb_b8e0_c9df7e43bd0f.slice/crio-6566e567d3d183707f7ee5753ca91345b65ba87361543afefdc82f0cf210d654 WatchSource:0}: Error finding container 6566e567d3d183707f7ee5753ca91345b65ba87361543afefdc82f0cf210d654: Status 404 returned error can't find the container with id 6566e567d3d183707f7ee5753ca91345b65ba87361543afefdc82f0cf210d654
Oct 30 20:18:53.906779 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:18:53.906758    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d15d48f_5197_4096_9011_d82bcb96fde9.slice/crio-0a784f7346dae1241766ef8d652276d345487ca8459a063e9f88b987c8ab8a99 WatchSource:0}: Error finding container 0a784f7346dae1241766ef8d652276d345487ca8459a063e9f88b987c8ab8a99: Status 404 returned error can't find the container with id 0a784f7346dae1241766ef8d652276d345487ca8459a063e9f88b987c8ab8a99
Oct 30 20:18:54.167561 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.167530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:54.167751 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.167727    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:54.167821 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.167809    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:18:55.167791847 +0000 UTC m=+4.126038096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:54.268706 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.268663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:54.268895 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.268827    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:54.268938 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.268899    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret podName:96f587e0-8984-434f-b1a5-bae1b56c343b nodeName:}" failed. No retries permitted until 2026-10-30 20:18:55.26887967 +0000 UTC m=+4.227125913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret") pod "global-pull-secret-syncer-jkspb" (UID: "96f587e0-8984-434f-b1a5-bae1b56c343b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:54.369723 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.369687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:54.369919 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.369840    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:18:54.369919 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.369855    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:18:54.369919 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.369866    2576 projected.go:196] Error preparing data for projected volume kube-api-access-tzwqg for pod openshift-network-diagnostics/network-check-target-n6f9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:54.369919 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:54.369919    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg podName:06d9b61a-283a-4ff0-b132-9319897c7e9b nodeName:}" failed. No retries permitted until 2026-10-30 20:18:55.369902063 +0000 UTC m=+4.328148304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzwqg" (UniqueName: "kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg") pod "network-check-target-n6f9w" (UID: "06d9b61a-283a-4ff0-b132-9319897c7e9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:54.603972 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.603917    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zz2fw" event={"ID":"7de73755-c298-441d-a185-bdaf04d2f7ea","Type":"ContainerStarted","Data":"beb2e1d8a32270e38bf7eba4a548f05a7072d7706406ec8a6ba2da857df41405"}
Oct 30 20:18:54.605464 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.605417    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58" event={"ID":"b93cda06-ee44-4b46-8a1b-1cee671fa4e7","Type":"ContainerStarted","Data":"3de02f77db8a82d890509743cb0efee95efbd63a22919932eedb4237664ef034"}
Oct 30 20:18:54.607333 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.607302    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"0a784f7346dae1241766ef8d652276d345487ca8459a063e9f88b987c8ab8a99"}
Oct 30 20:18:54.609031 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.609000    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x2lgj" event={"ID":"877333f1-3281-4c9a-84be-ac1de10369e4","Type":"ContainerStarted","Data":"b9fc7f744d19c4b75767b7635c4bfdb672ebd771bdfb2eaedce7d3f5d30151e7"}
Oct 30 20:18:54.610784 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.610757    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b5kfh" event={"ID":"f6244e4c-72f8-40ce-9a76-2afc5467cebe","Type":"ContainerStarted","Data":"926623d0c661fe33616eaeca447149441d5d295dc2dd90534c80533adb588b21"}
Oct 30 20:18:54.612302 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.612269    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerStarted","Data":"edfcc54666535d456059b2c9f25f78e0a3ab73ae0461ba5d9edd0809589b7c08"}
Oct 30 20:18:54.613935 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.613906    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l4ldp" event={"ID":"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f","Type":"ContainerStarted","Data":"6566e567d3d183707f7ee5753ca91345b65ba87361543afefdc82f0cf210d654"}
Oct 30 20:18:54.615342 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.615319    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nv9fr" event={"ID":"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109","Type":"ContainerStarted","Data":"beb5e664fd0e98dc0a7e5256f71f5c9df12431436100b8e0f3cc165d097264ec"}
Oct 30 20:18:54.617304 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.617270    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g6df8" event={"ID":"2b0926a1-74b6-4510-8fbc-91609d959cbf","Type":"ContainerStarted","Data":"c941c1c439823ba7453351ffba09a6ef0a3f801cfec417e7f7f5bdc7a639de3e"}
Oct 30 20:18:54.641555 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.641470    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:13:54 +0000 UTC" deadline="2028-06-18 14:30:15.414359361 +0000 UTC"
Oct 30 20:18:54.641555 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:54.641556    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14322h11m20.772822082s"
Oct 30 20:18:55.181461 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.181186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:55.181461 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.181354    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:55.181461 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.181420    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:18:57.181399614 +0000 UTC m=+6.139645860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:55.282540 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.282183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:55.282540 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.282378    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:55.282540 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.282450    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret podName:96f587e0-8984-434f-b1a5-bae1b56c343b nodeName:}" failed. No retries permitted until 2026-10-30 20:18:57.282429902 +0000 UTC m=+6.240676146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret") pod "global-pull-secret-syncer-jkspb" (UID: "96f587e0-8984-434f-b1a5-bae1b56c343b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:55.383993 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.383615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:55.383993 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.383828    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:18:55.383993 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.383856    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:18:55.383993 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.383870    2576 projected.go:196] Error preparing data for projected volume kube-api-access-tzwqg for pod openshift-network-diagnostics/network-check-target-n6f9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:55.383993 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.383932    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg podName:06d9b61a-283a-4ff0-b132-9319897c7e9b nodeName:}" failed. No retries permitted until 2026-10-30 20:18:57.383913833 +0000 UTC m=+6.342160077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzwqg" (UniqueName: "kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg") pod "network-check-target-n6f9w" (UID: "06d9b61a-283a-4ff0-b132-9319897c7e9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:55.606852 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.606261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:55.606852 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.606415    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:18:55.606852 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.606511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:55.606852 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.606616    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:18:55.606852 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.606678    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:55.606852 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:55.606765    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:18:55.641899 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.641841    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:13:54 +0000 UTC" deadline="2028-06-05 18:52:47.906657327 +0000 UTC"
Oct 30 20:18:55.641899 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:55.641887    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14014h33m52.264775081s"
Oct 30 20:18:57.209081 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:57.209043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:57.209619 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.209241    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:57.209619 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.209326    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:19:01.209304388 +0000 UTC m=+10.167550643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:18:57.310306 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:57.310199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:57.310494 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.310422    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:57.310494 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.310492    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret podName:96f587e0-8984-434f-b1a5-bae1b56c343b nodeName:}" failed. No retries permitted until 2026-10-30 20:19:01.310471858 +0000 UTC m=+10.268718122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret") pod "global-pull-secret-syncer-jkspb" (UID: "96f587e0-8984-434f-b1a5-bae1b56c343b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:18:57.411074 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:57.410985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:57.411246 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.411176    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:18:57.411246 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.411201    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:18:57.411246 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.411213    2576 projected.go:196] Error preparing data for projected volume kube-api-access-tzwqg for pod openshift-network-diagnostics/network-check-target-n6f9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:57.411477 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.411278    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg podName:06d9b61a-283a-4ff0-b132-9319897c7e9b nodeName:}" failed. No retries permitted until 2026-10-30 20:19:01.411259112 +0000 UTC m=+10.369505367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzwqg" (UniqueName: "kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg") pod "network-check-target-n6f9w" (UID: "06d9b61a-283a-4ff0-b132-9319897c7e9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:18:57.604501 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:57.604046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:57.604501 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.604183    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:18:57.604501 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:57.604275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:57.604501 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.604356    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:18:57.604501 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:57.604455    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:57.604799 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:57.604600    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:18:58.647138 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:58.646749    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"a97eddfe748e1e815e76819b9d90360d","Type":"ContainerStarted","Data":"69697f617293cf91c6a227759524249cd4e4b4a42f46d6ce50a0f546bae71312"}
Oct 30 20:18:58.652930 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:58.652756    2576 generic.go:356] "Generic (PLEG): container finished" podID="2203f4a44f682afa473e5863b4adcd60" containerID="1a7de360c89f4a57884ceefb797302ec4160434d8a82c9fda837951a36274bde" exitCode=0
Oct 30 20:18:58.652930 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:58.652870    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"2203f4a44f682afa473e5863b4adcd60","Type":"ContainerDied","Data":"1a7de360c89f4a57884ceefb797302ec4160434d8a82c9fda837951a36274bde"}
Oct 30 20:18:58.660959 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:58.660096    2576 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="df49543dc330e61bebb644a50c75d65bc561c09207fdae1fe0dda16c23de9c6d" exitCode=0
Oct 30 20:18:58.660959 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:58.660141    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerDied","Data":"df49543dc330e61bebb644a50c75d65bc561c09207fdae1fe0dda16c23de9c6d"}
Oct 30 20:18:58.771479 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:58.771376    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal" podStartSLOduration=5.771354132 podStartE2EDuration="5.771354132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:58.770607302 +0000 UTC m=+7.728853567" watchObservedRunningTime="2026-10-30 20:18:58.771354132 +0000 UTC m=+7.729600398"
Oct 30 20:18:59.596774 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.596704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:18:59.596959 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:59.596861    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:18:59.597325 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.597302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:18:59.597448 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:59.597425    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:18:59.597763 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.597740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:18:59.597880 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:18:59.597837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:18:59.670698 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.670459    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"2203f4a44f682afa473e5863b4adcd60","Type":"ContainerStarted","Data":"f36e46557a88cbe66299f6ff4f8051f4ff22470b719723ce2a3a96cce2f9b846"}
Oct 30 20:18:59.675354 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.675320    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l4ldp" event={"ID":"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f","Type":"ContainerStarted","Data":"1d6475c8e8608d29fd98ff3170ccbef13df114b69fdbc7c68aa6c63b78718866"}
Oct 30 20:18:59.802165 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.802093    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal" podStartSLOduration=6.802068731 podStartE2EDuration="6.802068731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:59.801132042 +0000 UTC m=+8.759378306" watchObservedRunningTime="2026-10-30 20:18:59.802068731 +0000 UTC m=+8.760314997"
Oct 30 20:18:59.826768 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.826725    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:59.827049 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.827027    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:18:59.929599 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:18:59.929465    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l4ldp" podStartSLOduration=1.993798962 podStartE2EDuration="6.929442009s" totalImagesPullingTime="4.935643047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:59.929017419 +0000 UTC m=+8.887263684" watchObservedRunningTime="2026-10-30 20:18:59.929442009 +0000 UTC m=+8.887688274"
Oct 30 20:19:00.686650 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:00.686559    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x2lgj" event={"ID":"877333f1-3281-4c9a-84be-ac1de10369e4","Type":"ContainerStarted","Data":"587a9c7987917a83c2e1b90ebb4c54a752f358cb8cd23a0d60a34fc774e28f51"}
Oct 30 20:19:01.249037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.248945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:01.249221 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.249107    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:01.249221 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.249174    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:19:09.24915536 +0000 UTC m=+18.207401615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:01.350122 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.350069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:01.350349 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.350278    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:01.350349 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.350346    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret podName:96f587e0-8984-434f-b1a5-bae1b56c343b nodeName:}" failed. No retries permitted until 2026-10-30 20:19:09.350324753 +0000 UTC m=+18.308571007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret") pod "global-pull-secret-syncer-jkspb" (UID: "96f587e0-8984-434f-b1a5-bae1b56c343b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:01.450891 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.450845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:01.451102 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.451016    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:01.451102 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.451045    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:01.451102 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.451060    2576 projected.go:196] Error preparing data for projected volume kube-api-access-tzwqg for pod openshift-network-diagnostics/network-check-target-n6f9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:01.451332 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.451126    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg podName:06d9b61a-283a-4ff0-b132-9319897c7e9b nodeName:}" failed. No retries permitted until 2026-10-30 20:19:09.451106376 +0000 UTC m=+18.409352631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzwqg" (UniqueName: "kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg") pod "network-check-target-n6f9w" (UID: "06d9b61a-283a-4ff0-b132-9319897c7e9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:01.595771 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.595675    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:01.595947 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.595845    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:01.597230 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.597200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:01.597332 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.597303    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:01.597370 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.597354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:01.597455 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:01.597435    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:01.693054 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.693020    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:01.693612 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:01.693560    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58" event={"ID":"b93cda06-ee44-4b46-8a1b-1cee671fa4e7","Type":"ContainerStarted","Data":"fca697114fa6b77b753d2617f6a1e69a159048aa18ae75d68eae549c1149c428"}
Oct 30 20:19:03.215389 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.215360    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:19:03.536134 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.535992    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:19:03.215383902Z","UUID":"1fda0e74-0e04-4f3e-8554-eb64ba50dbdd","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:19:03.538070 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.538040    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:19:03.538209 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.538083    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:19:03.599474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.599407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:03.599474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.599409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:03.599737 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:03.599600    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:03.599737 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.599648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:03.599815 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:03.599752    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:03.599853 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:03.599835    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:03.706200 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.706160    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b5kfh" event={"ID":"f6244e4c-72f8-40ce-9a76-2afc5467cebe","Type":"ContainerStarted","Data":"525a4297b53d9ca753a5c908157e200acf713abbe673c22994d7b2794aac3edb"}
Oct 30 20:19:03.709359 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.709314    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g6df8" event={"ID":"2b0926a1-74b6-4510-8fbc-91609d959cbf","Type":"ContainerStarted","Data":"e102cb70ca4baa91a9a4d36bb695889f9cac0b680b9518c2e8d5b824aef20a11"}
Oct 30 20:19:03.713270 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.713223    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58" event={"ID":"b93cda06-ee44-4b46-8a1b-1cee671fa4e7","Type":"ContainerStarted","Data":"006c9dac95383dbe40946417cd95035c2c95c8b8d1b194ad15c643d29281b396"}
Oct 30 20:19:03.828921 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.828651    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x2lgj" podStartSLOduration=4.521967716 podStartE2EDuration="10.828629265s" totalImagesPullingTime="6.306661549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:00.811648737 +0000 UTC m=+9.769895002" watchObservedRunningTime="2026-10-30 20:19:03.828629265 +0000 UTC m=+12.786875534"
Oct 30 20:19:03.953813 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.953000    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g6df8" podStartSLOduration=1.957468338 podStartE2EDuration="10.952980307s" totalImagesPullingTime="8.995511969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:03.95273922 +0000 UTC m=+12.910985483" watchObservedRunningTime="2026-10-30 20:19:03.952980307 +0000 UTC m=+12.911226571"
Oct 30 20:19:03.953813 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:03.953315    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-b5kfh" podStartSLOduration=1.417235515 podStartE2EDuration="10.953306385s" totalImagesPullingTime="9.53607087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:03.829133046 +0000 UTC m=+12.787379312" watchObservedRunningTime="2026-10-30 20:19:03.953306385 +0000 UTC m=+12.911552648"
Oct 30 20:19:04.719346 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:04.719303    2576 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="cddf0cc2d939edb61be54c9a42299b71c8f16032d0e330c17de659270e8eab2e" exitCode=0
Oct 30 20:19:04.719895 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:04.719651    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerDied","Data":"cddf0cc2d939edb61be54c9a42299b71c8f16032d0e330c17de659270e8eab2e"}
Oct 30 20:19:05.596188 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:05.596150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:05.596363 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:05.596160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:05.596363 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:05.596302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:05.596495 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:05.596271    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:05.596495 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:05.596392    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:05.596495 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:05.596430    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:05.723463 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:05.723379    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nv9fr" event={"ID":"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109","Type":"ContainerStarted","Data":"25475c525f08694b6115c999379a3c0acf59824a11d8e23cec9648090265acad"}
Oct 30 20:19:05.725016 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:05.724992    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zz2fw" event={"ID":"7de73755-c298-441d-a185-bdaf04d2f7ea","Type":"ContainerStarted","Data":"4ff05288a46f10244db136be7fccf5be55407e47347735a6554af593193f0a79"}
Oct 30 20:19:05.727019 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:05.726996    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58" event={"ID":"b93cda06-ee44-4b46-8a1b-1cee671fa4e7","Type":"ContainerStarted","Data":"07d495a1d92cb8570be46f797a854708ca30993ca37d3c4821d37bd39ede94a8"}
Oct 30 20:19:05.836677 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:05.836620    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nv9fr" podStartSLOduration=3.4433826610000002 podStartE2EDuration="12.836606154s" totalImagesPullingTime="9.393223493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:05.835915173 +0000 UTC m=+14.794161433" watchObservedRunningTime="2026-10-30 20:19:05.836606154 +0000 UTC m=+14.794852414"
Oct 30 20:19:06.067754 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:06.067701    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58" podStartSLOduration=2.457299175 podStartE2EDuration="13.067685941s" totalImagesPullingTime="10.610386766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:06.067598263 +0000 UTC m=+15.025844523" watchObservedRunningTime="2026-10-30 20:19:06.067685941 +0000 UTC m=+15.025932239"
Oct 30 20:19:06.067921 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:06.067811    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zz2fw" podStartSLOduration=1.4761294839999999 podStartE2EDuration="13.06780437s" totalImagesPullingTime="11.591674886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:05.950913958 +0000 UTC m=+14.909160229" watchObservedRunningTime="2026-10-30 20:19:06.06780437 +0000 UTC m=+15.026050630"
Oct 30 20:19:06.732452 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:06.732414    2576 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="748c1163e5aa38a19e75f1cf26a6ac28e1e5528701142a71bb3a4db6b79e36a0" exitCode=0
Oct 30 20:19:06.733133 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:06.732508    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerDied","Data":"748c1163e5aa38a19e75f1cf26a6ac28e1e5528701142a71bb3a4db6b79e36a0"}
Oct 30 20:19:07.188909 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:07.188813    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:19:07.189066 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:07.189003    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:07.189547 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:07.189507    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:19:07.596087 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:07.596011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:07.596087 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:07.596055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:07.596324 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:07.596017    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:07.596324 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:07.596154    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:07.596324 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:07.596256    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:07.596436 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:07.596345    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:08.740863 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:08.740818    2576 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="87153e04e25c6a9834bc568162fcda58ce4f662566ce73475ff5a756fbb4a39d" exitCode=0
Oct 30 20:19:08.741521 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:08.740908    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerDied","Data":"87153e04e25c6a9834bc568162fcda58ce4f662566ce73475ff5a756fbb4a39d"}
Oct 30 20:19:09.316613 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.316375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:09.316791 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.316554    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:09.316791 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.316684    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:19:25.31666751 +0000 UTC m=+34.274913748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:09.416958 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.416914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:09.417142 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.417070    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:09.417215 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.417144    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret podName:96f587e0-8984-434f-b1a5-bae1b56c343b nodeName:}" failed. No retries permitted until 2026-10-30 20:19:25.417122954 +0000 UTC m=+34.375369210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret") pod "global-pull-secret-syncer-jkspb" (UID: "96f587e0-8984-434f-b1a5-bae1b56c343b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:09.517819 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.517784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:09.517961 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.517937    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:09.517961 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.517958    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:09.518037 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.517967    2576 projected.go:196] Error preparing data for projected volume kube-api-access-tzwqg for pod openshift-network-diagnostics/network-check-target-n6f9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:09.518037 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.518024    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg podName:06d9b61a-283a-4ff0-b132-9319897c7e9b nodeName:}" failed. No retries permitted until 2026-10-30 20:19:25.518007195 +0000 UTC m=+34.476253433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzwqg" (UniqueName: "kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg") pod "network-check-target-n6f9w" (UID: "06d9b61a-283a-4ff0-b132-9319897c7e9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:09.598624 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.598523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:09.598624 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.598564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:09.598624 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.598522    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:09.598877 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.598672    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:09.598877 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.598725    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:09.598877 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:09.598788    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:09.747521 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.747489    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"3e7046d699a873b10397fd5d3a6f3d7f542b738db33ed30204f189dac4687779"}
Oct 30 20:19:09.747973 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:09.747581    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"3c80b5a661e87dfbdcb21a85db97ea32ab3601aab2eb0e93f1d6abc826d07576"}
Oct 30 20:19:10.752631 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:10.752425    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"38fc069745b43d9cb4d31e5baa1c265c1f130bc4835a3dcf3c4a06dd769d0911"}
Oct 30 20:19:10.753058 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:10.752716    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"01149d22136faa3b72650749c3e423a210a2c56151213ea3e658e21307d22395"}
Oct 30 20:19:10.753058 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:10.752772    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"42d8b46de71b5df8e8845758fa3c1a7a9fdea5ba995e031c641471c9237e89fd"}
Oct 30 20:19:10.753058 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:10.752823    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"6eacb39c5410ae06a5699058595f85a295b4e9742f157a2963e86b5ac1c0f82f"}
Oct 30 20:19:11.596774 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:11.596736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:11.596935 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:11.596880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:11.596935 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:11.596913    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:11.597034 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:11.596946    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:11.597034 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:11.596985    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:11.597096 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:11.597075    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:11.757197 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:11.757164    2576 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="d78c3ac22c7a6920dc72361db31a2239b4edb8234497d620b3407a7d8fb85312" exitCode=0
Oct 30 20:19:11.757754 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:11.757222    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerDied","Data":"d78c3ac22c7a6920dc72361db31a2239b4edb8234497d620b3407a7d8fb85312"}
Oct 30 20:19:12.763226 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:12.763184    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"39cdca77c83f82e0b426457927edc575b00fdaa3234fd9627ddb2492135e73f8"}
Oct 30 20:19:12.766177 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:12.766145    2576 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="a77a2717ef98ef53c722fc7570efc4da2e35e020f7e19cb157583ed58f8a40a0" exitCode=0
Oct 30 20:19:12.766307 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:12.766204    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerDied","Data":"a77a2717ef98ef53c722fc7570efc4da2e35e020f7e19cb157583ed58f8a40a0"}
Oct 30 20:19:13.595289 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:13.595263    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:13.595289 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:13.595288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:13.595493 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:13.595380    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:13.595493 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:13.595421    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:13.595493 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:13.595485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:13.595667 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:13.595601    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:13.771731 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:13.771694    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r25dm" event={"ID":"62ad0a1a-eede-46f0-8e64-88b9d86a5239","Type":"ContainerStarted","Data":"885c8686864801d5fad4b6ba8fa6c19575458dec207214cb9b1a2eb85a18cf14"}
Oct 30 20:19:14.780069 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:14.779731    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" event={"ID":"2d15d48f-5197-4096-9011-d82bcb96fde9","Type":"ContainerStarted","Data":"e8b98d6bf36fac5a9fa6f03065959bdfc4a8c0f491c1a58fb4614185363c7526"}
Oct 30 20:19:15.055037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.054947    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r25dm" podStartSLOduration=9.450989478 podStartE2EDuration="22.054932352s" totalImagesPullingTime="12.603942874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:13.957745013 +0000 UTC m=+22.915991273" watchObservedRunningTime="2026-10-30 20:19:15.054932352 +0000 UTC m=+24.013178612"
Oct 30 20:19:15.055202 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.055124    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk" podStartSLOduration=6.359654166 podStartE2EDuration="22.055116973s" totalImagesPullingTime="15.695462807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:15.054682627 +0000 UTC m=+24.012928887" watchObservedRunningTime="2026-10-30 20:19:15.055116973 +0000 UTC m=+24.013363418"
Oct 30 20:19:15.596168 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.596131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:15.596168 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.596152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:15.596410 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:15.596240    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:15.596410 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.596292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:15.596487 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:15.596422    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:15.596567 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:15.596521    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:15.782524 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.782492    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:15.782972 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.782598    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:15.782972 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.782642    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:15.798510 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.798480    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:15.798737 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:15.798645    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:17.207495 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:17.207460    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6f9w"]
Oct 30 20:19:17.208000 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:17.207643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:17.208000 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:17.207757    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:17.214663 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:17.214641    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkspb"]
Oct 30 20:19:17.214792 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:17.214737    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:17.214846 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:17.214832    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:17.215404 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:17.215380    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svcz9"]
Oct 30 20:19:17.215502 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:17.215492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:17.215620 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:17.215598    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:18.595895 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:18.595865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:18.596338 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:18.595960    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:18.596338 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:18.596052    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:18.596338 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:18.596183    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:19.595970 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:19.595900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:19.596427 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:19.596005    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:20.596006 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:20.595979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:20.596006 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:20.595998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:20.596471 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:20.596081    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:20.596471 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:20.596233    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:21.597248 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:21.597220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:21.597642 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:21.597303    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkspb" podUID="96f587e0-8984-434f-b1a5-bae1b56c343b"
Oct 30 20:19:22.595380 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:22.595356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:22.595380 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:22.595388    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:22.595592 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:22.595465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svcz9" podUID="3084f525-41f6-44e7-9cf4-e0a85e23362e"
Oct 30 20:19:22.595625 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:22.595607    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6f9w" podUID="06d9b61a-283a-4ff0-b132-9319897c7e9b"
Oct 30 20:19:22.893779 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:22.893596    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:19:22.894170 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:22.893893    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:19:22.979214 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:22.979181    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nr6ns"]
Oct 30 20:19:22.981088 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:22.981071    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nr6ns"
Oct 30 20:19:23.007906 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.007883    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"]
Oct 30 20:19:23.009946 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.009921    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dqz7k"]
Oct 30 20:19:23.010114 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.010093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.011224 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.011208    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-df452"]
Oct 30 20:19:23.011354 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.011333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dqz7k"
Oct 30 20:19:23.012482 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.012468    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mjg55"]
Oct 30 20:19:23.012623 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.012608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-df452"
Oct 30 20:19:23.013739 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.013726    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67cdd9f98c-zf64k"]
Oct 30 20:19:23.013844 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.013826    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.019447 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.019431    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"]
Oct 30 20:19:23.019591 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.019576    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.020683 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.020668    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"]
Oct 30 20:19:23.020804 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.020789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.022183 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.022163    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"]
Oct 30 20:19:23.022267 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.022247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"
Oct 30 20:19:23.023485 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.023472    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"]
Oct 30 20:19:23.023619 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.023595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.024954 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.024935    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-695dcdfbc7-z5fr7"]
Oct 30 20:19:23.025080 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.025066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.026403 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.026387    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"]
Oct 30 20:19:23.026493 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.026478    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.027748 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.027734    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.028713 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.028695    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f4544dfb8-94842"]
Oct 30 20:19:23.030127 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.030110    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"]
Oct 30 20:19:23.030229 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.030213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.031459 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.031444    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"]
Oct 30 20:19:23.031596 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.031581    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"
Oct 30 20:19:23.033209 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.033191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:23.056641 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.056620    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.056726 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.056647    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.057478 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.057461    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8hmbp"]
Oct 30 20:19:23.059012 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.058996    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nnc2v"]
Oct 30 20:19:23.059129 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.059115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.060553 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.060523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.061812 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.061792    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 20:19:23.061940 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.061927    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q5fsc\""
Oct 30 20:19:23.069322 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.069305    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nr6ns"]
Oct 30 20:19:23.074378 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.074358    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"]
Oct 30 20:19:23.075771 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.075489    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dqz7k"]
Oct 30 20:19:23.078914 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.078892    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mjg55"]
Oct 30 20:19:23.097885 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.097862    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"]
Oct 30 20:19:23.098671 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.098654    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67cdd9f98c-zf64k"]
Oct 30 20:19:23.099389 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.099370    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4544dfb8-94842"]
Oct 30 20:19:23.099961 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.099937    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-695dcdfbc7-z5fr7"]
Oct 30 20:19:23.100648 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.100630    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"]
Oct 30 20:19:23.116501 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.116603 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh4kb\" (UniqueName: \"kubernetes.io/projected/7b7f4d68-a578-434f-894a-6cf2fbc63cda-kube-api-access-bh4kb\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.116603 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jkpt\" (UniqueName: \"kubernetes.io/projected/70698df1-1b21-49a3-9cb1-131b2b113024-kube-api-access-5jkpt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.116708 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.116763 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61c90821-34fa-4790-a80c-34e6fb6e02ce-serving-cert\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.116763 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfntl\" (UniqueName: \"kubernetes.io/projected/b85f4408-b319-46f6-aef0-87b5efd91eec-kube-api-access-jfntl\") pod \"kas-connection-checker-685895c7c9-dqz7k\" (UID: \"b85f4408-b319-46f6-aef0-87b5efd91eec\") " pod="kube-system/kas-connection-checker-685895c7c9-dqz7k"
Oct 30 20:19:23.116868 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-service-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.116868 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvph\" (UniqueName: \"kubernetes.io/projected/61c90821-34fa-4790-a80c-34e6fb6e02ce-kube-api-access-wwvph\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.116944 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.116981 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.116953    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.117026 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.117062 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9dm\" (UniqueName: \"kubernetes.io/projected/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d-kube-api-access-zb9dm\") pod \"volume-data-source-validator-58c9f5f46d-x2hmz\" (UID: \"ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"
Oct 30 20:19:23.117107 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.117107 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-config\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.117216 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.117216 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-default-certificate\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.117216 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.117377 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtd6\" (UniqueName: \"kubernetes.io/projected/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-kube-api-access-7wtd6\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.117377 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.117508 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.117508 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkds\" (UniqueName: \"kubernetes.io/projected/f8932125-9eb4-4f7f-b577-c232a34a0520-kube-api-access-fxkds\") pod \"network-check-source-6f78d59f99-fbzx6\" (UID: \"f8932125-9eb4-4f7f-b577-c232a34a0520\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"
Oct 30 20:19:23.117508 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.117508 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117475    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.117508 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6j59\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.117781 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxk6\" (UniqueName: \"kubernetes.io/projected/c2d3fa2f-6503-48f1-a830-375faa8c3a38-kube-api-access-jcxk6\") pod \"kas-connection-checker-685895c7c9-nr6ns\" (UID: \"c2d3fa2f-6503-48f1-a830-375faa8c3a38\") " pod="kube-system/kas-connection-checker-685895c7c9-nr6ns"
Oct 30 20:19:23.117781 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.117781 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-trusted-ca\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.117781 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70698df1-1b21-49a3-9cb1-131b2b113024-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.117781 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974nn\" (UniqueName: \"kubernetes.io/projected/31fce4ed-e652-4919-9cae-157650ca980f-kube-api-access-974nn\") pod \"kas-connection-checker-685895c7c9-df452\" (UID: \"31fce4ed-e652-4919-9cae-157650ca980f\") " pod="kube-system/kas-connection-checker-685895c7c9-df452"
Oct 30 20:19:23.118010 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xs7r\" (UniqueName: \"kubernetes.io/projected/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-kube-api-access-5xs7r\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.118010 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-snapshots\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.118010 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70698df1-1b21-49a3-9cb1-131b2b113024-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.118010 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:23.118010 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.117978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.118010 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ceacc798-3509-453a-b3ad-48abc179967b-serving-cert\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-stats-auth\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-tmp\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71f15b82-a100-493b-aef0-c83d07ad1806-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7f4d68-a578-434f-894a-6cf2fbc63cda-serving-cert\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.118319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.118766 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wcbv\" (UniqueName: \"kubernetes.io/projected/ceacc798-3509-453a-b3ad-48abc179967b-kube-api-access-7wcbv\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.118766 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.118766 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ceacc798-3509-453a-b3ad-48abc179967b-config\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.118766 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.118766 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sc4x\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.118766 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.118710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tfv\" (UniqueName: \"kubernetes.io/projected/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-kube-api-access-78tfv\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.119336 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.119311    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nnc2v"]
Oct 30 20:19:23.130474 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.130457    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-df452"]
Oct 30 20:19:23.133315 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133284    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.133315 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133292    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 20:19:23.133477 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133452    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 20:19:23.133590 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133573    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.133697 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133684    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.133767 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133757    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.133821 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133783    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.133821 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133806    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.133922 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133846    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:19:23.133922 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133866    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.133922 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133876    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.133922 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133909    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.133922 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133917    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.134139 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133931    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.134139 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.133582    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.134139 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134028    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 20:19:23.134139 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134115    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.134139 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134119    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.134139 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134133    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134175    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134178    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134115    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134208    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134220    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134248    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134290    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134326    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 20:19:23.134390 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.134390    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.183554 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.183476    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 20:19:23.185060 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.185034    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"]
Oct 30 20:19:23.190893 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.190875    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:19:23.191136 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.191118    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:19:23.191273 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.191245    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 20:19:23.191376 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.191315    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 20:19:23.191376 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.191330    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:19:23.191491 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.191478    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ckn8f\""
Oct 30 20:19:23.192110 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.192093    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hmbp"]
Oct 30 20:19:23.192574 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.192561    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 20:19:23.192648 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.192635    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:19:23.192689 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.192651    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"]
Oct 30 20:19:23.196943 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.196920    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 20:19:23.197028 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.196948    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:19:23.197028 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.196951    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kd6m6\""
Oct 30 20:19:23.197028 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.196952    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vdqtm\""
Oct 30 20:19:23.197028 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.196981    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fb4b4\""
Oct 30 20:19:23.216232 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.216211    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 20:19:23.218826 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.218811    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hbnvb\""
Oct 30 20:19:23.218989 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.218975    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:19:23.219046 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.218993    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q6sjr\""
Oct 30 20:19:23.219046 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219018    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 20:19:23.219046 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219026    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 20:19:23.219046 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.218998    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 20:19:23.219046 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219046    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4tp44\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219053    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219075    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rnpmh\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219095    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219110    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219135    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l7r4r\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4t9\" (UniqueName: \"kubernetes.io/projected/a4840983-85d4-48ae-8de5-20aa5de9f42c-kube-api-access-ww4t9\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974nn\" (UniqueName: \"kubernetes.io/projected/31fce4ed-e652-4919-9cae-157650ca980f-kube-api-access-974nn\") pod \"kas-connection-checker-685895c7c9-df452\" (UID: \"31fce4ed-e652-4919-9cae-157650ca980f\") " pod="kube-system/kas-connection-checker-685895c7c9-df452"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219223    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bp5wk\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xs7r\" (UniqueName: \"kubernetes.io/projected/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-kube-api-access-5xs7r\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-snapshots\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70698df1-1b21-49a3-9cb1-131b2b113024-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219347    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6vlmd\""
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ceacc798-3509-453a-b3ad-48abc179967b-serving-cert\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-stats-auth\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d32d84-47e1-4b90-83e1-3509cd867f17-config-volume\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.219443 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-tmp\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71f15b82-a100-493b-aef0-c83d07ad1806-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7f4d68-a578-434f-894a-6cf2fbc63cda-serving-cert\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcbv\" (UniqueName: \"kubernetes.io/projected/ceacc798-3509-453a-b3ad-48abc179967b-kube-api-access-7wcbv\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ceacc798-3509-453a-b3ad-48abc179967b-config\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sc4x\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tfv\" (UniqueName: \"kubernetes.io/projected/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-kube-api-access-78tfv\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.220283 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh4kb\" (UniqueName: \"kubernetes.io/projected/7b7f4d68-a578-434f-894a-6cf2fbc63cda-kube-api-access-bh4kb\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.219951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkpt\" (UniqueName: \"kubernetes.io/projected/70698df1-1b21-49a3-9cb1-131b2b113024-kube-api-access-5jkpt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.220000    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70698df1-1b21-49a3-9cb1-131b2b113024-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.220070    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.720051218 +0000 UTC m=+32.678297474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-snapshots\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61c90821-34fa-4790-a80c-34e6fb6e02ce-serving-cert\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfntl\" (UniqueName: \"kubernetes.io/projected/b85f4408-b319-46f6-aef0-87b5efd91eec-kube-api-access-jfntl\") pod \"kas-connection-checker-685895c7c9-dqz7k\" (UID: \"b85f4408-b319-46f6-aef0-87b5efd91eec\") " pod="kube-system/kas-connection-checker-685895c7c9-dqz7k"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-service-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71f15b82-a100-493b-aef0-c83d07ad1806-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.221015 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.220620    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.221488 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.221109    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ceacc798-3509-453a-b3ad-48abc179967b-config\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.221488 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.221182    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.221488 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.221210    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:23.221488 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.221223    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:23.221488 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.221264    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.721250753 +0000 UTC m=+32.679496995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:23.222449 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222367    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.222720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvph\" (UniqueName: \"kubernetes.io/projected/61c90821-34fa-4790-a80c-34e6fb6e02ce-kube-api-access-wwvph\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.222720 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.222846 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222723    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-tmp\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.222846 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.222846 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.222846 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9dm\" (UniqueName: \"kubernetes.io/projected/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d-kube-api-access-zb9dm\") pod \"volume-data-source-validator-58c9f5f46d-x2hmz\" (UID: \"ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"
Oct 30 20:19:23.223040 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.223040 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-config\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.223040 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.223040 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-default-certificate\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.223040 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.223040 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.223000    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.223040 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.223375 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.223052    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:23.223375 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.223059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtd6\" (UniqueName: \"kubernetes.io/projected/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-kube-api-access-7wtd6\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.223375 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.223093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d32d84-47e1-4b90-83e1-3509cd867f17-tmp-dir\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.223375 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.222916    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"]
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.223863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.223933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.223969    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkds\" (UniqueName: \"kubernetes.io/projected/f8932125-9eb4-4f7f-b577-c232a34a0520-kube-api-access-fxkds\") pod \"network-check-source-6f78d59f99-fbzx6\" (UID: \"f8932125-9eb4-4f7f-b577-c232a34a0520\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6j59\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxk6\" (UniqueName: \"kubernetes.io/projected/c2d3fa2f-6503-48f1-a830-375faa8c3a38-kube-api-access-jcxk6\") pod \"kas-connection-checker-685895c7c9-nr6ns\" (UID: \"c2d3fa2f-6503-48f1-a830-375faa8c3a38\") " pod="kube-system/kas-connection-checker-685895c7c9-nr6ns"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-trusted-ca\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70698df1-1b21-49a3-9cb1-131b2b113024-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9z9\" (UniqueName: \"kubernetes.io/projected/c6d32d84-47e1-4b90-83e1-3509cd867f17-kube-api-access-7s9z9\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4840983-85d4-48ae-8de5-20aa5de9f42c-cert\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.224569    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.724551306 +0000 UTC m=+32.682797546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.224899    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-service-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.226664 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.225127    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-config\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.227386 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.225524    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.227386 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.225601    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"]
Oct 30 20:19:23.227386 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.225644    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"]
Oct 30 20:19:23.227386 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.227054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ceacc798-3509-453a-b3ad-48abc179967b-serving-cert\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.227386 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.227269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-stats-auth\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.227386 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.227380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.227682 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.227428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61c90821-34fa-4790-a80c-34e6fb6e02ce-serving-cert\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.227730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.228002    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.727985761 +0000 UTC m=+32.686232021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.228081    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.228129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7f4d68-a578-434f-894a-6cf2fbc63cda-serving-cert\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.228346    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.228375    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.228457    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.728440418 +0000 UTC m=+32.686686674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.228480    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.228557    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.728523064 +0000 UTC m=+32.686769320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.228726    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.228798    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:23.228856 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.228820    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:23.229433 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.229022    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.728998689 +0000 UTC m=+32.687244941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:23.229433 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.229335    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-trusted-ca\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.231627 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.231506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70698df1-1b21-49a3-9cb1-131b2b113024-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.232245 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.232221    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.232399 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.232380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.232946 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.232924    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-default-certificate\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.276431 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.276408    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974nn\" (UniqueName: \"kubernetes.io/projected/31fce4ed-e652-4919-9cae-157650ca980f-kube-api-access-974nn\") pod \"kas-connection-checker-685895c7c9-df452\" (UID: \"31fce4ed-e652-4919-9cae-157650ca980f\") " pod="kube-system/kas-connection-checker-685895c7c9-df452"
Oct 30 20:19:23.276928 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.276908    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xs7r\" (UniqueName: \"kubernetes.io/projected/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-kube-api-access-5xs7r\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.279634 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.279611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.281083 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.281035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tfv\" (UniqueName: \"kubernetes.io/projected/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-kube-api-access-78tfv\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.281278 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.281200    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh4kb\" (UniqueName: \"kubernetes.io/projected/7b7f4d68-a578-434f-894a-6cf2fbc63cda-kube-api-access-bh4kb\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.281278 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.281222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sc4x\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.281504 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.281484    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.288092 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.288065    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcbv\" (UniqueName: \"kubernetes.io/projected/ceacc798-3509-453a-b3ad-48abc179967b-kube-api-access-7wcbv\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.291231 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.291171    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6j59\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.291564 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.291513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkpt\" (UniqueName: \"kubernetes.io/projected/70698df1-1b21-49a3-9cb1-131b2b113024-kube-api-access-5jkpt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.291648 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.291605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxk6\" (UniqueName: \"kubernetes.io/projected/c2d3fa2f-6503-48f1-a830-375faa8c3a38-kube-api-access-jcxk6\") pod \"kas-connection-checker-685895c7c9-nr6ns\" (UID: \"c2d3fa2f-6503-48f1-a830-375faa8c3a38\") " pod="kube-system/kas-connection-checker-685895c7c9-nr6ns"
Oct 30 20:19:23.292183 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.292159    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9dm\" (UniqueName: \"kubernetes.io/projected/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d-kube-api-access-zb9dm\") pod \"volume-data-source-validator-58c9f5f46d-x2hmz\" (UID: \"ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"
Oct 30 20:19:23.292183 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.292177    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtd6\" (UniqueName: \"kubernetes.io/projected/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-kube-api-access-7wtd6\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.292351 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.292232    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvph\" (UniqueName: \"kubernetes.io/projected/61c90821-34fa-4790-a80c-34e6fb6e02ce-kube-api-access-wwvph\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.292783 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.292757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfntl\" (UniqueName: \"kubernetes.io/projected/b85f4408-b319-46f6-aef0-87b5efd91eec-kube-api-access-jfntl\") pod \"kas-connection-checker-685895c7c9-dqz7k\" (UID: \"b85f4408-b319-46f6-aef0-87b5efd91eec\") " pod="kube-system/kas-connection-checker-685895c7c9-dqz7k"
Oct 30 20:19:23.293551 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.293516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkds\" (UniqueName: \"kubernetes.io/projected/f8932125-9eb4-4f7f-b577-c232a34a0520-kube-api-access-fxkds\") pod \"network-check-source-6f78d59f99-fbzx6\" (UID: \"f8932125-9eb4-4f7f-b577-c232a34a0520\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"
Oct 30 20:19:23.327087 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d32d84-47e1-4b90-83e1-3509cd867f17-tmp-dir\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.327193 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9z9\" (UniqueName: \"kubernetes.io/projected/c6d32d84-47e1-4b90-83e1-3509cd867f17-kube-api-access-7s9z9\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.327252 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4840983-85d4-48ae-8de5-20aa5de9f42c-cert\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.327317 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4t9\" (UniqueName: \"kubernetes.io/projected/a4840983-85d4-48ae-8de5-20aa5de9f42c-kube-api-access-ww4t9\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.327317 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d32d84-47e1-4b90-83e1-3509cd867f17-config-volume\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.327420 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.327501 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327473    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d32d84-47e1-4b90-83e1-3509cd867f17-tmp-dir\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.327581 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.327510    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:23.327581 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.327580    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:23.827568523 +0000 UTC m=+32.785814760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:23.327844 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.327817    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d32d84-47e1-4b90-83e1-3509cd867f17-config-volume\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.328755 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.328738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dqz7k"
Oct 30 20:19:23.329523 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.329503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4840983-85d4-48ae-8de5-20aa5de9f42c-cert\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.335712 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.335691    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-df452"
Oct 30 20:19:23.341629 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.341612    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:23.353037 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.352905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:23.357725 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.357685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"
Oct 30 20:19:23.372950 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.372757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:23.385780 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.385753    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9z9\" (UniqueName: \"kubernetes.io/projected/c6d32d84-47e1-4b90-83e1-3509cd867f17-kube-api-access-7s9z9\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.389301 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.389021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:23.390684 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.390605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4t9\" (UniqueName: \"kubernetes.io/projected/a4840983-85d4-48ae-8de5-20aa5de9f42c-kube-api-access-ww4t9\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.413230 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.408863    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"
Oct 30 20:19:23.423591 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.423566    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:23.520879 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.520783    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:23.591907 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.591887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nr6ns"
Oct 30 20:19:23.596240 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.596221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:23.648651 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.648626    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wz6nt\""
Oct 30 20:19:23.648762 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.648686    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:19:23.725053 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.724688    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-df452"]
Oct 30 20:19:23.727442 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.727421    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dqz7k"]
Oct 30 20:19:23.728030 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.728004    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fce4ed_e652_4919_9cae_157650ca980f.slice/crio-c8a40a08560c8a5aa382406d83ac44aa536b1c02f40f06f8b1dd91dfc4ab6ce0 WatchSource:0}: Error finding container c8a40a08560c8a5aa382406d83ac44aa536b1c02f40f06f8b1dd91dfc4ab6ce0: Status 404 returned error can't find the container with id c8a40a08560c8a5aa382406d83ac44aa536b1c02f40f06f8b1dd91dfc4ab6ce0
Oct 30 20:19:23.729958 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.729933    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85f4408_b319_46f6_aef0_87b5efd91eec.slice/crio-24008d4ce6ed626d6dd444051831bb0d08ca2022ef6a95ad4d09186871841c91 WatchSource:0}: Error finding container 24008d4ce6ed626d6dd444051831bb0d08ca2022ef6a95ad4d09186871841c91: Status 404 returned error can't find the container with id 24008d4ce6ed626d6dd444051831bb0d08ca2022ef6a95ad4d09186871841c91
Oct 30 20:19:23.731115 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.731092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:23.731219 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.731141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:23.731219 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.731205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.731341 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.731228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:23.731341 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.731254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:23.731341 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731232    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:23.731341 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731290    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:23.731341 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.731305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:23.731341 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731313    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.731333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731350    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.73132923 +0000 UTC m=+33.689575481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731286    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731371    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731378    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.731367691 +0000 UTC m=+33.689613929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731396    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.731386916 +0000 UTC m=+33.689633169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731410    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731417    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.731402291 +0000 UTC m=+33.689648551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731420    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731421    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731458    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.73143745 +0000 UTC m=+33.689683699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731487    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.731475607 +0000 UTC m=+33.689721849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:23.732052 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.731507    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.73149764 +0000 UTC m=+33.689743881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:23.782765 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.782735    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mjg55"]
Oct 30 20:19:23.788456 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.788234    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c90821_34fa_4790_a80c_34e6fb6e02ce.slice/crio-798cd14fdb96c4e8c773936493eb9d83844d0cdbc4632928f5cd5c60b94a3d8c WatchSource:0}: Error finding container 798cd14fdb96c4e8c773936493eb9d83844d0cdbc4632928f5cd5c60b94a3d8c: Status 404 returned error can't find the container with id 798cd14fdb96c4e8c773936493eb9d83844d0cdbc4632928f5cd5c60b94a3d8c
Oct 30 20:19:23.790868 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.790795    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"]
Oct 30 20:19:23.791886 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.791860    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"]
Oct 30 20:19:23.798320 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.798276    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70698df1_1b21_49a3_9cb1_131b2b113024.slice/crio-6253727e873eed1a0617db25d3a223df4c0f4f0a6cefc5779e0f3a24dc419ae8 WatchSource:0}: Error finding container 6253727e873eed1a0617db25d3a223df4c0f4f0a6cefc5779e0f3a24dc419ae8: Status 404 returned error can't find the container with id 6253727e873eed1a0617db25d3a223df4c0f4f0a6cefc5779e0f3a24dc419ae8
Oct 30 20:19:23.801466 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.801444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:23.806148 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.806127    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-df452" event={"ID":"31fce4ed-e652-4919-9cae-157650ca980f","Type":"ContainerStarted","Data":"c8a40a08560c8a5aa382406d83ac44aa536b1c02f40f06f8b1dd91dfc4ab6ce0"}
Oct 30 20:19:23.807585 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.807558    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mjg55" event={"ID":"61c90821-34fa-4790-a80c-34e6fb6e02ce","Type":"ContainerStarted","Data":"798cd14fdb96c4e8c773936493eb9d83844d0cdbc4632928f5cd5c60b94a3d8c"}
Oct 30 20:19:23.808756 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.808737    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dqz7k" event={"ID":"b85f4408-b319-46f6-aef0-87b5efd91eec","Type":"ContainerStarted","Data":"24008d4ce6ed626d6dd444051831bb0d08ca2022ef6a95ad4d09186871841c91"}
Oct 30 20:19:23.813417 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.813399    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"]
Oct 30 20:19:23.819336 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.819315    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7f4d68_a578_434f_894a_6cf2fbc63cda.slice/crio-140e14d58453a7fbe1994328bde86a8e48131cbe3dc77c1ee84ac9a85383f2fb WatchSource:0}: Error finding container 140e14d58453a7fbe1994328bde86a8e48131cbe3dc77c1ee84ac9a85383f2fb: Status 404 returned error can't find the container with id 140e14d58453a7fbe1994328bde86a8e48131cbe3dc77c1ee84ac9a85383f2fb
Oct 30 20:19:23.828446 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.828427    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"]
Oct 30 20:19:23.832028 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.832010    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:23.832177 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.832159    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:23.832245 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:23.832219    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.832204576 +0000 UTC m=+33.790450814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:23.854985 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.854962    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nnc2v"]
Oct 30 20:19:23.859057 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.859028    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4840983_85d4_48ae_8de5_20aa5de9f42c.slice/crio-e76075a4085499ba41418da6dfa8bc4eab47352b6980ff99e56a7579ebce5d32 WatchSource:0}: Error finding container e76075a4085499ba41418da6dfa8bc4eab47352b6980ff99e56a7579ebce5d32: Status 404 returned error can't find the container with id e76075a4085499ba41418da6dfa8bc4eab47352b6980ff99e56a7579ebce5d32
Oct 30 20:19:23.859246 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.859229    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"]
Oct 30 20:19:23.862014 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.861978    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8932125_9eb4_4f7f_b577_c232a34a0520.slice/crio-1421c65a896b1a21f81006414f1720bb7e71282e2332f279c1f906aa94e019ea WatchSource:0}: Error finding container 1421c65a896b1a21f81006414f1720bb7e71282e2332f279c1f906aa94e019ea: Status 404 returned error can't find the container with id 1421c65a896b1a21f81006414f1720bb7e71282e2332f279c1f906aa94e019ea
Oct 30 20:19:23.957127 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:23.957105    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nr6ns"]
Oct 30 20:19:23.960321 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:23.960295    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d3fa2f_6503_48f1_a830_375faa8c3a38.slice/crio-104d02079ce754ff33e427702f69874ea14666d2db848ab0cde16e05d78600b2 WatchSource:0}: Error finding container 104d02079ce754ff33e427702f69874ea14666d2db848ab0cde16e05d78600b2: Status 404 returned error can't find the container with id 104d02079ce754ff33e427702f69874ea14666d2db848ab0cde16e05d78600b2
Oct 30 20:19:24.595424 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.595389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:24.595613 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.595563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:24.647136 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.647110    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:19:24.647245 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.647230    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxq5f\""
Oct 30 20:19:24.647319 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.647287    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nnssb\""
Oct 30 20:19:24.740913 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.740877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:24.741076 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.740957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:24.741076 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741033    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:24.741193 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741101    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.741082017 +0000 UTC m=+35.699328258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:24.741193 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741117    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:24.741193 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741135    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:24.741354 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741196    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.741178199 +0000 UTC m=+35.699424448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:24.741354 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.741222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:24.741354 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.741262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:24.741354 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.741295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:24.741354 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741335    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.74132205 +0000 UTC m=+35.699568292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741377    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.741382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741411    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.741400175 +0000 UTC m=+35.699646416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.741429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741458    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741474    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741493    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.741481588 +0000 UTC m=+35.699727826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741523    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741557    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741557    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.741517781 +0000 UTC m=+35.699764032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:24.741786 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.741599    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.741587944 +0000 UTC m=+35.699834189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:24.816264 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.816232    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" event={"ID":"7b7f4d68-a578-434f-894a-6cf2fbc63cda","Type":"ContainerStarted","Data":"140e14d58453a7fbe1994328bde86a8e48131cbe3dc77c1ee84ac9a85383f2fb"}
Oct 30 20:19:24.819307 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.819279    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6" event={"ID":"f8932125-9eb4-4f7f-b577-c232a34a0520","Type":"ContainerStarted","Data":"1421c65a896b1a21f81006414f1720bb7e71282e2332f279c1f906aa94e019ea"}
Oct 30 20:19:24.824395 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.824368    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7" event={"ID":"70698df1-1b21-49a3-9cb1-131b2b113024","Type":"ContainerStarted","Data":"6253727e873eed1a0617db25d3a223df4c0f4f0a6cefc5779e0f3a24dc419ae8"}
Oct 30 20:19:24.828173 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.828151    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-df452" event={"ID":"31fce4ed-e652-4919-9cae-157650ca980f","Type":"ContainerStarted","Data":"dc039b3d9da3e7da4aab79212dd1933cdc4bf613858c14cfc9979027af7f733a"}
Oct 30 20:19:24.830699 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.830256    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dqz7k" event={"ID":"b85f4408-b319-46f6-aef0-87b5efd91eec","Type":"ContainerStarted","Data":"31b924707929807b03a37f02550e21579cfbf830a28980ec92fb1b8336cbea62"}
Oct 30 20:19:24.832409 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.832343    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn" event={"ID":"ceacc798-3509-453a-b3ad-48abc179967b","Type":"ContainerStarted","Data":"8ea3da950d87b1423934d69c0b2b95bf7057f33aaa382e1d200f8269229b9c0a"}
Oct 30 20:19:24.834058 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.833990    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz" event={"ID":"ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d","Type":"ContainerStarted","Data":"9d8e93a4469889ffb7b33be3d5838737c48d16ae2325ab5acb425a6cbf1ac314"}
Oct 30 20:19:24.836034 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.835610    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nr6ns" event={"ID":"c2d3fa2f-6503-48f1-a830-375faa8c3a38","Type":"ContainerStarted","Data":"428978b30a6d8dd77cd6de2237ba0a2efc1d66222be4c0144b6dea3ec3071fa7"}
Oct 30 20:19:24.836034 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.835654    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nr6ns" event={"ID":"c2d3fa2f-6503-48f1-a830-375faa8c3a38","Type":"ContainerStarted","Data":"104d02079ce754ff33e427702f69874ea14666d2db848ab0cde16e05d78600b2"}
Oct 30 20:19:24.838458 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.838435    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nnc2v" event={"ID":"a4840983-85d4-48ae-8de5-20aa5de9f42c","Type":"ContainerStarted","Data":"e76075a4085499ba41418da6dfa8bc4eab47352b6980ff99e56a7579ebce5d32"}
Oct 30 20:19:24.841966 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:24.841947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:24.842102 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.842084    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:24.842171 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:24.842148    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:26.842130159 +0000 UTC m=+35.800376420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:25.271036 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.270952    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nr6ns" podStartSLOduration=389.270914831 podStartE2EDuration="6m29.270914831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:25.157514436 +0000 UTC m=+34.115760700" watchObservedRunningTime="2026-10-30 20:19:25.270914831 +0000 UTC m=+34.229161092"
Oct 30 20:19:25.273791 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.272454    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dqz7k" podStartSLOduration=389.272444249 podStartE2EDuration="6m29.272444249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:25.270447189 +0000 UTC m=+34.228693449" watchObservedRunningTime="2026-10-30 20:19:25.272444249 +0000 UTC m=+34.230690510"
Oct 30 20:19:25.347297 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.347255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:25.347556 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:25.347518    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:19:25.347656 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:25.347628    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:19:57.347608613 +0000 UTC m=+66.305854852 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : secret "metrics-daemon-secret" not found
Oct 30 20:19:25.385232 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.383700    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-df452" podStartSLOduration=389.383688033 podStartE2EDuration="6m29.383688033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:25.383103082 +0000 UTC m=+34.341349369" watchObservedRunningTime="2026-10-30 20:19:25.383688033 +0000 UTC m=+34.341934296"
Oct 30 20:19:25.449293 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.448618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:25.452346 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.452264    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:25.550026 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.549827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:25.555933 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.555863    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:25.707397 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.707369    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:25.806768 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:25.806277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:26.110296 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.110188    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkspb"]
Oct 30 20:19:26.219858 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.219776    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6f9w"]
Oct 30 20:19:26.230524 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:26.230447    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d9b61a_283a_4ff0_b132_9319897c7e9b.slice/crio-2caf2490dde9e6bca18f861979d838c9b1abfed9ef9744274c1e66c6ac12f6a6 WatchSource:0}: Error finding container 2caf2490dde9e6bca18f861979d838c9b1abfed9ef9744274c1e66c6ac12f6a6: Status 404 returned error can't find the container with id 2caf2490dde9e6bca18f861979d838c9b1abfed9ef9744274c1e66c6ac12f6a6
Oct 30 20:19:26.763373 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.763337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.763378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.763404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.763455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.763499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.763570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.763621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.763786    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:26.763834 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.763802    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:26.764196 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.763869    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.763849751 +0000 UTC m=+39.722096008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764303    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.764284627 +0000 UTC m=+39.722530879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764382    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764420    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.764406368 +0000 UTC m=+39.722652609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764474    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764502    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.764493138 +0000 UTC m=+39.722739395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764574    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764605    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.764595599 +0000 UTC m=+39.722841852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764661    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764670    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764695    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.764682066 +0000 UTC m=+39.722928317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764730    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:26.764797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.764748    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.764741987 +0000 UTC m=+39.722988238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:26.864940 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.863672    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkspb" event={"ID":"96f587e0-8984-434f-b1a5-bae1b56c343b","Type":"ContainerStarted","Data":"8d7573a0ee248b4183fbc5ae615a5eadc929fb47464dc58c3b5f5a30dae5958c"}
Oct 30 20:19:26.864940 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.864362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:26.864940 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.864675    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:26.864940 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:26.864742    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.864721386 +0000 UTC m=+39.822967645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:26.866675 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.866569    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6f9w" event={"ID":"06d9b61a-283a-4ff0-b132-9319897c7e9b","Type":"ContainerStarted","Data":"2caf2490dde9e6bca18f861979d838c9b1abfed9ef9744274c1e66c6ac12f6a6"}
Oct 30 20:19:26.875200 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.875171    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz" event={"ID":"ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d","Type":"ContainerStarted","Data":"ec112013ddeeb459a7fef5b2b26ccde057b40c09efc37bf31f1d7a9945354653"}
Oct 30 20:19:26.994072 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:26.993986    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz" podStartSLOduration=362.549403794 podStartE2EDuration="6m4.993968143s" totalImagesPullingTime="2.444564349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:26.992527423 +0000 UTC m=+35.950773681" watchObservedRunningTime="2026-10-30 20:19:26.993968143 +0000 UTC m=+35.952214405"
Oct 30 20:19:27.152858 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:27.152773    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:27.387402 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:27.387336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:28.890418 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:28.890353    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/0.log"
Oct 30 20:19:28.890418 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:28.890388    2576 generic.go:356] "Generic (PLEG): container finished" podID="7b7f4d68-a578-434f-894a-6cf2fbc63cda" containerID="8b9cbb3fae83eb811e159e742ab584251bd167b3cd2243ed1e3721a287777a58" exitCode=255
Oct 30 20:19:28.890952 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:28.890463    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" event={"ID":"7b7f4d68-a578-434f-894a-6cf2fbc63cda","Type":"ContainerDied","Data":"8b9cbb3fae83eb811e159e742ab584251bd167b3cd2243ed1e3721a287777a58"}
Oct 30 20:19:28.890952 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:28.890823    2576 scope.go:122] "RemoveContainer" containerID="8b9cbb3fae83eb811e159e742ab584251bd167b3cd2243ed1e3721a287777a58"
Oct 30 20:19:28.893879 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:28.893412    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7" event={"ID":"70698df1-1b21-49a3-9cb1-131b2b113024","Type":"ContainerStarted","Data":"4de31d38d5970e8ab7991252cd1146b339371b3b1d89ed9ae5de1f87023938c7"}
Oct 30 20:19:28.897359 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:28.897255    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mjg55" event={"ID":"61c90821-34fa-4790-a80c-34e6fb6e02ce","Type":"ContainerStarted","Data":"80dcc95db37a5d404f48f5c075ee1b63491012c968f5f8afe048df7e64b9f25b"}
Oct 30 20:19:29.143985 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.143898    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7" podStartSLOduration=362.557259505 podStartE2EDuration="6m7.143879125s" totalImagesPullingTime="4.58661962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:29.141911556 +0000 UTC m=+38.100157845" watchObservedRunningTime="2026-10-30 20:19:29.143879125 +0000 UTC m=+38.102125390"
Oct 30 20:19:29.271342 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.270828    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-mjg55" podStartSLOduration=362.697811226 podStartE2EDuration="6m7.270808115s" totalImagesPullingTime="4.572996889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:29.266908436 +0000 UTC m=+38.225154698" watchObservedRunningTime="2026-10-30 20:19:29.270808115 +0000 UTC m=+38.229054381"
Oct 30 20:19:29.920118 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.920018    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn" event={"ID":"ceacc798-3509-453a-b3ad-48abc179967b","Type":"ContainerStarted","Data":"855403074bd9b34a97d983347bc51a9f339502a475f13f6fadef37ce9e3b4cde"}
Oct 30 20:19:29.923249 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.923159    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/1.log"
Oct 30 20:19:29.925242 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.923736    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/0.log"
Oct 30 20:19:29.925242 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.923769    2576 generic.go:356] "Generic (PLEG): container finished" podID="7b7f4d68-a578-434f-894a-6cf2fbc63cda" containerID="6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b" exitCode=255
Oct 30 20:19:29.925242 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.924375    2576 scope.go:122] "RemoveContainer" containerID="6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b"
Oct 30 20:19:29.925242 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:29.924558    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jnzz2_openshift-console-operator(7b7f4d68-a578-434f-894a-6cf2fbc63cda)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" podUID="7b7f4d68-a578-434f-894a-6cf2fbc63cda"
Oct 30 20:19:29.925242 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.924726    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" event={"ID":"7b7f4d68-a578-434f-894a-6cf2fbc63cda","Type":"ContainerDied","Data":"6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b"}
Oct 30 20:19:29.925242 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:29.924775    2576 scope.go:122] "RemoveContainer" containerID="8b9cbb3fae83eb811e159e742ab584251bd167b3cd2243ed1e3721a287777a58"
Oct 30 20:19:30.054888 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.054667    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn" podStartSLOduration=362.891423374 podStartE2EDuration="6m8.054648802s" totalImagesPullingTime="5.163225428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:30.052520116 +0000 UTC m=+39.010766377" watchObservedRunningTime="2026-10-30 20:19:30.054648802 +0000 UTC m=+39.012895061"
Oct 30 20:19:30.098263 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.096203    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cmnm9"]
Oct 30 20:19:30.100424 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.099610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.120257 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.120235    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:30.159980 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.159639    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:19:30.159980 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.159728    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:19:30.160645 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.160621    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4ct2\""
Oct 30 20:19:30.163576 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.162448    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cmnm9"]
Oct 30 20:19:30.199697 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.199628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpkt\" (UniqueName: \"kubernetes.io/projected/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-api-access-snpkt\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.199697 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.199691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-crio-socket\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.199894 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.199822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.199894 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.199861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.200002 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.199902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-data-volume\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.223383 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.223355    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"]
Oct 30 20:19:30.227636 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.226723    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"
Oct 30 20:19:30.234376 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.233200    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"]
Oct 30 20:19:30.237057 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.236796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"
Oct 30 20:19:30.283645 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.283619    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h8r9r\""
Oct 30 20:19:30.286933 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.286867    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"]
Oct 30 20:19:30.298759 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.298701    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"]
Oct 30 20:19:30.301894 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.301724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2md7\" (UniqueName: \"kubernetes.io/projected/c9906391-b55c-4c9c-bb13-58b6339d1f76-kube-api-access-h2md7\") pod \"migrator-66df5d496b-d6p67\" (UID: \"c9906391-b55c-4c9c-bb13-58b6339d1f76\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"
Oct 30 20:19:30.302190 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.302167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.302264 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.302217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.302264 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.302248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdvv\" (UniqueName: \"kubernetes.io/projected/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f-kube-api-access-tpdvv\") pod \"migrator-66df5d496b-qlgnw\" (UID: \"d34f102b-c1f4-4e19-a7ce-cdc1ea51930f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"
Oct 30 20:19:30.302348 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.302286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-data-volume\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.302348 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.302338    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:30.302422 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.302357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpkt\" (UniqueName: \"kubernetes.io/projected/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-api-access-snpkt\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.302422 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.302405    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.802384854 +0000 UTC m=+39.760631106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:30.302498 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.302441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-crio-socket\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.302601 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.302584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-crio-socket\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.302797 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.302769    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:30.302860 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.302846    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:30.802824619 +0000 UTC m=+39.761070874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:30.304203 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.304177    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-data-volume\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.363242 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.361490    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:19:30.363242 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.361576    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:30.364292 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.364259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:30.366869 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.366851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpkt\" (UniqueName: \"kubernetes.io/projected/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-api-access-snpkt\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.403438 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.403336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdvv\" (UniqueName: \"kubernetes.io/projected/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f-kube-api-access-tpdvv\") pod \"migrator-66df5d496b-qlgnw\" (UID: \"d34f102b-c1f4-4e19-a7ce-cdc1ea51930f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"
Oct 30 20:19:30.403641 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.403485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2md7\" (UniqueName: \"kubernetes.io/projected/c9906391-b55c-4c9c-bb13-58b6339d1f76-kube-api-access-h2md7\") pod \"migrator-66df5d496b-d6p67\" (UID: \"c9906391-b55c-4c9c-bb13-58b6339d1f76\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"
Oct 30 20:19:30.464128 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.464096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2md7\" (UniqueName: \"kubernetes.io/projected/c9906391-b55c-4c9c-bb13-58b6339d1f76-kube-api-access-h2md7\") pod \"migrator-66df5d496b-d6p67\" (UID: \"c9906391-b55c-4c9c-bb13-58b6339d1f76\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"
Oct 30 20:19:30.465576 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.465550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdvv\" (UniqueName: \"kubernetes.io/projected/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f-kube-api-access-tpdvv\") pod \"migrator-66df5d496b-qlgnw\" (UID: \"d34f102b-c1f4-4e19-a7ce-cdc1ea51930f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"
Oct 30 20:19:30.548413 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.548381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"
Oct 30 20:19:30.556304 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.556275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"
Oct 30 20:19:30.568354 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.568327    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.807649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.807782    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.807798    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.807866    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.807847523 +0000 UTC m=+47.766093767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808249    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808306    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.808288744 +0000 UTC m=+47.766534985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808373    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:30.808851 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808386    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808417    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.808407008 +0000 UTC m=+47.766653259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808579    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808662    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808699    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808744    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.808825    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.809568    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.808474663 +0000 UTC m=+47.766720916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.809607    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:31.809591274 +0000 UTC m=+40.767837518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.809631    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.809621042 +0000 UTC m=+47.767867528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.809645    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.809636831 +0000 UTC m=+47.767883076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.809661    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.809652832 +0000 UTC m=+47.767899072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:30.809754 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.809676    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:31.809667716 +0000 UTC m=+40.767913958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:30.881715 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.881619    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:30.908960 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.908932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:30.909236 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.909215    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:30.909350 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.909286    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.90926615 +0000 UTC m=+47.867512388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:30.929869 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.929842    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/1.log"
Oct 30 20:19:30.930372 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.930345    2576 scope.go:122] "RemoveContainer" containerID="6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b"
Oct 30 20:19:30.930599 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:30.930573    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jnzz2_openshift-console-operator(7b7f4d68-a578-434f-894a-6cf2fbc63cda)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" podUID="7b7f4d68-a578-434f-894a-6cf2fbc63cda"
Oct 30 20:19:30.932079 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.931936    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6" event={"ID":"f8932125-9eb4-4f7f-b577-c232a34a0520","Type":"ContainerStarted","Data":"3479c495133b0933d6742b903bb44aea7982a3400d23bdbe7997257f5af0c68e"}
Oct 30 20:19:30.934509 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.934412    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nnc2v" event={"ID":"a4840983-85d4-48ae-8de5-20aa5de9f42c","Type":"ContainerStarted","Data":"2694e9d686ccdd028f9e0c37c7e299ce5c15f1affca1feb7391e863c4826437c"}
Oct 30 20:19:30.964036 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:30.963976    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"]
Oct 30 20:19:30.964036 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:30.963997    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34f102b_c1f4_4e19_a7ce_cdc1ea51930f.slice/crio-146a6b5d870e6775e806055b841d849b94c7248fd43818abe2dde7641442397c WatchSource:0}: Error finding container 146a6b5d870e6775e806055b841d849b94c7248fd43818abe2dde7641442397c: Status 404 returned error can't find the container with id 146a6b5d870e6775e806055b841d849b94c7248fd43818abe2dde7641442397c
Oct 30 20:19:31.004102 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.003884    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"]
Oct 30 20:19:31.006891 ip-10-0-141-29 kubenswrapper[2576]: W1030 20:19:31.006860    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9906391_b55c_4c9c_bb13_58b6339d1f76.slice/crio-d94adc8d6a13bba4e7e21f9fb208b17d97d0b305c8538e3fce7749331c5c1a3e WatchSource:0}: Error finding container d94adc8d6a13bba4e7e21f9fb208b17d97d0b305c8538e3fce7749331c5c1a3e: Status 404 returned error can't find the container with id d94adc8d6a13bba4e7e21f9fb208b17d97d0b305c8538e3fce7749331c5c1a3e
Oct 30 20:19:31.157392 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.157253    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:31.177991 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.177934    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nnc2v" podStartSLOduration=3.046338712 podStartE2EDuration="9.177914815s" totalImagesPullingTime="6.131576103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:31.176760344 +0000 UTC m=+40.135006607" watchObservedRunningTime="2026-10-30 20:19:31.177914815 +0000 UTC m=+40.136161087"
Oct 30 20:19:31.292217 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.292169    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6" podStartSLOduration=358.584825142 podStartE2EDuration="6m5.292155295s" totalImagesPullingTime="6.707330153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:31.291270919 +0000 UTC m=+40.249517182" watchObservedRunningTime="2026-10-30 20:19:31.292155295 +0000 UTC m=+40.250401552"
Oct 30 20:19:31.817349 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.817313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:31.817467 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.817361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:31.817724 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:31.817570    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:31.817724 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:31.817601    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:31.817724 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:31.817647    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:33.817627643 +0000 UTC m=+42.775873896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:31.817724 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:31.817661    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:33.817654876 +0000 UTC m=+42.775901114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:31.940150 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.940113    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67" event={"ID":"c9906391-b55c-4c9c-bb13-58b6339d1f76","Type":"ContainerStarted","Data":"d94adc8d6a13bba4e7e21f9fb208b17d97d0b305c8538e3fce7749331c5c1a3e"}
Oct 30 20:19:31.941590 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.941551    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw" event={"ID":"d34f102b-c1f4-4e19-a7ce-cdc1ea51930f","Type":"ContainerStarted","Data":"146a6b5d870e6775e806055b841d849b94c7248fd43818abe2dde7641442397c"}
Oct 30 20:19:31.943231 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.943202    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6f9w" event={"ID":"06d9b61a-283a-4ff0-b132-9319897c7e9b","Type":"ContainerStarted","Data":"1e6b42e5eb58bffbb1de7ab3fa192722afbcf3635d857fa8500265bd838d55e5"}
Oct 30 20:19:31.960985 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:31.960952    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:32.224044 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:32.224006    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:32.424420 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:32.424318    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:32.722386 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:32.722353    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:32.948566 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:32.948507    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67" event={"ID":"c9906391-b55c-4c9c-bb13-58b6339d1f76","Type":"ContainerStarted","Data":"e7cd93a9cfc1aa1dc28d14ee4c066e79e17d2c8b76ee2680d07a63ee81f89cbc"}
Oct 30 20:19:32.950284 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:32.950249    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw" event={"ID":"d34f102b-c1f4-4e19-a7ce-cdc1ea51930f","Type":"ContainerStarted","Data":"ea6abed99fe687a0de5db24203574b85536978b50d47241c82ab267116a84f6a"}
Oct 30 20:19:32.950408 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:32.950362    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:33.063078 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:33.062984    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n6f9w" podStartSLOduration=34.481640192 podStartE2EDuration="40.062969168s" totalImagesPullingTime="5.581328976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:32.058344352 +0000 UTC m=+41.016590606" watchObservedRunningTime="2026-10-30 20:19:33.062969168 +0000 UTC m=+42.021215429"
Oct 30 20:19:33.178194 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:33.178142    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw" podStartSLOduration=1.359099667 podStartE2EDuration="3.178127893s" totalImagesPullingTime="1.819028226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:33.177031512 +0000 UTC m=+42.135277775" watchObservedRunningTime="2026-10-30 20:19:33.178127893 +0000 UTC m=+42.136374180"
Oct 30 20:19:33.178356 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:33.178334    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67" podStartSLOduration=1.4043062499999999 podStartE2EDuration="3.17832739s" totalImagesPullingTime="1.77402114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:33.064370381 +0000 UTC m=+42.022616640" watchObservedRunningTime="2026-10-30 20:19:33.17832739 +0000 UTC m=+42.136573713"
Oct 30 20:19:33.390001 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:33.389924    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:33.390001 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:33.389981    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:33.390345 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:33.390330    2576 scope.go:122] "RemoveContainer" containerID="6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b"
Oct 30 20:19:33.390579 ip-10-0-141-29 kubenswrapper[2576]: E1030 20:19:33.390554    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jnzz2_openshift-console-operator(7b7f4d68-a578-434f-894a-6cf2fbc63cda)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" podUID="7b7f4d68-a578-434f-894a-6cf2fbc63cda"
Oct 30 20:19:33.687488 ip-10-0-141-29 kubenswrapper[2576]: I1030 20:19:33.687393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:33.794735 ip-10-0-141-29 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:19:33.803200 ip-10-0-141-29 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:19:33.803364 ip-10-0-141-29 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:19:33.803417 ip-10-0-141-29 systemd[1]: kubelet.service: Consumed 3.264s CPU time.
Oct 30 20:19:33.820117 ip-10-0-141-29 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:19:33.880347 ip-10-0-141-29 kubenswrapper[5002]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:33.880347 ip-10-0-141-29 kubenswrapper[5002]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:19:33.880347 ip-10-0-141-29 kubenswrapper[5002]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:33.880347 ip-10-0-141-29 kubenswrapper[5002]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:33.882493 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882423    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.882493 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882494    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882498    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882502    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882505    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882508    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882510    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882513    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882515    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882518    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882520    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882523    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882526    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882545    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882551    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882555    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882559    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882562    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882564    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882574    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.882600 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882577    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882580    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882582    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882585    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882588    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882591    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882594    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882597    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882599    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882602    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882605    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882609    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882612    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882614    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882617    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882619    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882622    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882625    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882627    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882631    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.883068 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882634    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882637    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882639    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882642    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882644    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882647    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882649    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882652    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882654    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882657    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882659    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882662    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882665    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882667    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882670    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882672    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882675    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882677    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882680    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.883618 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882682    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882685    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882688    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882691    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882694    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882696    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882698    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882701    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882703    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882706    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882708    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882710    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882713    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882716    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882718    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882721    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882723    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882726    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882728    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882731    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.884081 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882734    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882736    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882738    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882742    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882745    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882747    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882750    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882752    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882754    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882757    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882759    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882762    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882764    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882766    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882770    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882773    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882775    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882778    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882780    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882783    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.884574 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882785    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882789    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882793    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882796    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882798    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882801    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.882804    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883574    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883581    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883584    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883588    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883590    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883593    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883595    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883599    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883603    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883607    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883610    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.885099 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883613    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883617    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883619    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883622    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883625    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883627    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883630    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883633    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883635    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883638    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883640    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883643    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883645    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883648    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883650    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883653    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883655    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883658    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883660    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883662    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.885569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883665    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883669    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883672    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883674    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883677    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883679    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883681    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883684    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883686    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883689    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883691    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883693    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883696    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883698    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883701    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883703    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883706    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883708    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883711    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.886072 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883713    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883716    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883718    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883721    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883724    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883726    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883729    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883732    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883735    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883737    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883740    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883742    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883745    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883747    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883749    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883752    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883755    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883757    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883760    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883762    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.886559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883764    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883767    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883769    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883771    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883774    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883777    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883781    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883785    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883788    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883791    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883794    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883797    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883800    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883803    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883806    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883808    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883811    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883813    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883815    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.887077 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883818    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883820    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883823    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883825    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883827    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883830    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883832    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883834    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883837    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883840    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883843    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883845    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883848    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883850    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883853    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883856    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.883858    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884326    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884331    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884334    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.887559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884340    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884343    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884345    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884348    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884350    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884353    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884355    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884358    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884360    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884363    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884365    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884368    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884371    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884373    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884376    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884378    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884380    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884383    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884385    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884388    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.888067 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884390    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884393    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884396    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884398    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884400    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884403    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884405    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884408    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884412    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884415    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884419    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884423    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884428    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884431    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884433    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884436    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884438    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884441    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884443    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.888569 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884446    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884448    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884451    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884453    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884456    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884459    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884462    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884465    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884467    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884470    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884472    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884475    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884477    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884480    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884482    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884486    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884488    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884491    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884493    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884495    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.889041 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884498    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884502    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884504    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884507    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884509    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884513    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884515    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884518    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884520    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884522    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884525    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884546    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884550    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884553    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884556    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884558    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884561    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884563    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884566    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884568    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.889559 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884571    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884573    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884575    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884578    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884581    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884584    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884587    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884589    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884592    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884595    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884597    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884600    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884603    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884607    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884609    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884611    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884614    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884617    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884620    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.890042 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884622    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884625    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884627    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884630    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.884633    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884730    5002 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884745    5002 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884750    5002 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884754    5002 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884758    5002 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884762    5002 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884766    5002 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884770    5002 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884773    5002 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884776    5002 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884779    5002 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884783    5002 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884786    5002 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884789    5002 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884792    5002 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884794    5002 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884797    5002 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884801    5002 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:19:33.890729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884806    5002 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884809    5002 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884812    5002 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884816    5002 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884820    5002 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884824    5002 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884826    5002 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884830    5002 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884833    5002 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884836    5002 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884839    5002 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884851    5002 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884855    5002 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884858    5002 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884862    5002 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884865    5002 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884868    5002 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884870    5002 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884873    5002 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884876    5002 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884880    5002 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884883    5002 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884886    5002 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884889    5002 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:19:33.891579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884892    5002 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884896    5002 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884899    5002 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884902    5002 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884904    5002 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884908    5002 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884911    5002 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884914    5002 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884917    5002 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884920    5002 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884923    5002 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884929    5002 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884933    5002 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884936    5002 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884939    5002 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884942    5002 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884946    5002 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884949    5002 flags.go:64] FLAG: --help="false"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884951    5002 flags.go:64] FLAG: --hostname-override="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884954    5002 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884958    5002 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884960    5002 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884964    5002 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:19:33.892165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884967    5002 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884970    5002 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884973    5002 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884976    5002 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884978    5002 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884981    5002 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884984    5002 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884987    5002 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884990    5002 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884992    5002 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884996    5002 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.884999    5002 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885001    5002 flags.go:64] FLAG: --lock-file=""
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885004    5002 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885007    5002 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885010    5002 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885015    5002 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885018    5002 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885027    5002 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885030    5002 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885034    5002 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885037    5002 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885040    5002 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885043    5002 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885047    5002 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:19:33.892743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885051    5002 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885054    5002 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885057    5002 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885060    5002 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885063    5002 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885066    5002 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885069    5002 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885071    5002 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885074    5002 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885081    5002 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885084    5002 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885087    5002 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885090    5002 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885093    5002 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885096    5002 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885099    5002 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885102    5002 flags.go:64] FLAG: --port="10250"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885105    5002 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885108    5002 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-07b37be0b914f8226"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885111    5002 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885114    5002 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885117    5002 flags.go:64] FLAG: --register-node="true"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885120    5002 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885123    5002 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885126    5002 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885130    5002 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:19:33.893368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885136    5002 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885141    5002 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885144    5002 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885147    5002 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885150    5002 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885153    5002 flags.go:64] FLAG: --runonce="false"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885155    5002 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885158    5002 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885161    5002 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885164    5002 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885167    5002 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885170    5002 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885172    5002 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885176    5002 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885178    5002 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885181    5002 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885184    5002 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885187    5002 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885190    5002 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885193    5002 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885195    5002 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885200    5002 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885203    5002 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885206    5002 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885210    5002 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:19:33.894107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885213    5002 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885215    5002 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885218    5002 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885221    5002 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885224    5002 flags.go:64] FLAG: --v="2"
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885228    5002 flags.go:64] FLAG: --version="false"
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885233    5002 flags.go:64] FLAG: --vmodule=""
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885236    5002 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885242    5002 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885420    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885428    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885434    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885439    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885444    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885449    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885454    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885459    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885463    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885467    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885472    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885476    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.894756 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885481    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885485    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885490    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885495    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885500    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885504    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885509    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885512    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885517    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885522    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885526    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885547    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885551    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885555    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885560    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885564    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885568    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885576    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885580    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.895485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885584    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885588    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885591    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885594    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885597    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885599    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885602    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885604    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885607    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885609    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885612    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885614    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885617    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885620    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885622    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885624    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885627    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885629    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885632    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.896079 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885634    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885638    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885642    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885644    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885647    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885649    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885652    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885654    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885657    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885659    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885662    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885666    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885668    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885671    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885674    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885677    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885679    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885682    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885684    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885687    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.896683 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885689    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885692    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885694    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885697    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885699    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885702    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885704    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885706    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885709    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885711    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885713    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885716    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885720    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885723    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885726    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885729    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885732    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885734    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885737    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.897213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885739    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885742    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885744    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885747    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885751    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885753    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885756    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885758    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885760    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885763    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885766    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885768    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885770    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885773    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885775    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885778    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.897777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885780    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.885807    5002 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885841    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885844    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885847    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885850    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885852    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885854    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885857    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885859    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885862    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885865    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885868    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885870    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885873    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885875    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.898266 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885878    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885880    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885883    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885885    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885889    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885892    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885894    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885897    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885899    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885901    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885904    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885907    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885909    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885912    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885914    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885917    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885919    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885922    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885925    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.898745 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885929    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885932    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885935    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885938    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885940    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885943    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885945    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885948    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885950    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885953    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885955    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885958    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885960    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885962    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885965    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885967    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885969    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885973    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885975    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885978    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.899205 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885980    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885983    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885985    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885988    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885990    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885993    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885995    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.885998    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886000    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886002    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886006    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886009    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886013    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886016    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886019    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886022    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886026    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886031    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886033    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.899766 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886036    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886039    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886041    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886043    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886046    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886048    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886051    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886053    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886055    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886058    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886061    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886064    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886066    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886068    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886071    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886074    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886076    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886079    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886090    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.900214 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886093    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886095    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886098    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886100    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886103    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886105    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886108    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886110    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886113    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886115    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886118    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886121    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886125    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886128    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.886130    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.889756    5002 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.889771    5002 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.889788    5002 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.889793    5002 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889853    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.900694 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889858    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889861    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889863    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889866    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889869    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889872    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889874    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889877    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889879    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889882    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889884    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889887    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889890    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889893    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889896    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889899    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889901    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889904    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889906    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.901306 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889909    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889911    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889914    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889916    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889919    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889921    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889924    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889928    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889931    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889933    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889936    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889939    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889941    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889943    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889946    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889948    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889951    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889953    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889956    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.901983 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889960    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889964    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889967    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889970    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889972    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889974    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889977    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889980    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889982    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889984    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889987    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889989    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889992    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889994    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889997    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.889999    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890001    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890004    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890006    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890009    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.902726 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890012    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890015    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890017    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890020    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890022    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890025    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890027    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890029    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890032    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890034    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890037    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890039    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890041    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890044    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890047    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890049    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890052    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890055    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890057    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890060    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.903241 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890062    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890065    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890067    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890070    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890072    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890074    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890077    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890079    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890082    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890084    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890087    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890091    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890093    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890096    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890098    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890101    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890103    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890105    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890108    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890110    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.903822 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890113    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890115    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890117    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890121    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890125    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890128    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890131    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.890158    5002 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890651    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890674    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890681    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890686    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890692    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890698    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890710    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.904299 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890719    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890724    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890728    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890733    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890737    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890741    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890746    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890750    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890754    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890759    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890763    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890769    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890778    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890783    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890787    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890792    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890796    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890800    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890804    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890809    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.904670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890814    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890819    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890823    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890828    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890832    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890842    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890849    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890854    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890858    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890863    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890867    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890871    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890878    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890882    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890887    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890892    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890897    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890906    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890910    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890915    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.905158 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890920    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890924    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890929    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890934    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890938    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890942    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890947    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890951    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890956    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890960    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890970    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890974    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890979    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890983    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890987    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.890994    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891000    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891005    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891010    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.905688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891016    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891021    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891026    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891035    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891040    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891047    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891051    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891055    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891060    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891064    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891069    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891074    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891078    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891082    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891087    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891092    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891102    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891106    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891111    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891115    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.906551 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891120    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891124    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891193    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891199    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891202    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891206    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891213    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891218    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891223    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891227    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891232    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891236    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891240    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891265    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891287    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891389    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891398    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891402    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891406    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.907147 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891410    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.891458    5002 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891511    5002 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891518    5002 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891523    5002 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891551    5002 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891556    5002 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891561    5002 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891564    5002 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891568    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891572    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891576    5002 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891581    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891585    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891589    5002 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891594    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:33.907660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891598    5002 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891602    5002 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891607    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891611    5002 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891615    5002 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891619    5002 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891623    5002 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891628    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891635    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891639    5002 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891643    5002 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891648    5002 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891653    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891657    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891660    5002 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891692    5002 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891696    5002 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891699    5002 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891701    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891704    5002 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:33.908076 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891706    5002 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891709    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891712    5002 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891714    5002 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891717    5002 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891719    5002 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891722    5002 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891724    5002 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891729    5002 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891733    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891736    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891739    5002 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891742    5002 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891745    5002 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891748    5002 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891750    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891753    5002 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891755    5002 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891757    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:33.908573 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891760    5002 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891763    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891766    5002 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891768    5002 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891771    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891773    5002 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891776    5002 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891778    5002 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891781    5002 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891783    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891786    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891788    5002 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891791    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891793    5002 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891796    5002 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891798    5002 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891841    5002 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891844    5002 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891853    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891857    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:33.909045 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891862    5002 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891866    5002 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891870    5002 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891873    5002 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891877    5002 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891882    5002 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891886    5002 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891890    5002 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891894    5002 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891897    5002 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891901    5002 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891907    5002 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891912    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891917    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891921    5002 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891925    5002 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891930    5002 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891933    5002 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891937    5002 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891941    5002 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:33.909713 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891945    5002 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891948    5002 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891951    5002 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891954    5002 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891956    5002 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891959    5002 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891962    5002 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891964    5002 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891967    5002 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891969    5002 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891974    5002 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891978    5002 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:33.891983    5002 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.892047    5002 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.892977    5002 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.893027    5002 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.893212    5002 server.go:1014] "Starting client certificate rotation"
Oct 30 20:19:33.910398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.893365    5002 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.893421    5002 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:13:52 +0000 UTC" deadline="2028-06-13 14:07:30.914536047 +0000 UTC"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.893463    5002 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14201h47m57.021076686s"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.893711    5002 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.894713    5002 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.895906    5002 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.897462    5002 log.go:25] "Validated CRI v1 image API"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.898172    5002 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:19:33.911166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.899955    5002 fs.go:135] Filesystem UUIDs: map[2aaa55d8-3651-47e0-be96-149c87065615:/dev/nvme0n1p4 41468992-4b9f-4a5e-8c12-4cd808a87e30:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 20:19:33.911677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.899974    5002 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a784f7346dae1241766ef8d652276d345487ca8459a063e9f88b987c8ab8a99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a784f7346dae1241766ef8d652276d345487ca8459a063e9f88b987c8ab8a99/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/104d02079ce754ff33e427702f69874ea14666d2db848ab0cde16e05d78600b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/104d02079ce754ff33e427702f69874ea14666d2db848ab0cde16e05d78600b2/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/140e14d58453a7fbe1994328bde86a8e48131cbe3dc77c1ee84ac9a85383f2fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/140e14d58453a7fbe1994328bde86a8e48131cbe3dc77c1ee84ac9a85383f2fb/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1421c65a896b1a21f81006414f1720bb7e71282e2332f279c1f906aa94e019ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1421c65a896b1a21f81006414f1720bb7e71282e2332f279c1f906aa94e019ea/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/146a6b5d870e6775e806055b841d849b94c7248fd43818abe2dde7641442397c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/146a6b5d870e6775e806055b841d849b94c7248fd43818abe2dde7641442397c/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24008d4ce6ed626d6dd444051831bb0d08ca2022ef6a95ad4d09186871841c91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24008d4ce6ed626d6dd444051831bb0d08ca2022ef6a95ad4d09186871841c91/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2caf2490dde9e6bca18f861979d838c9b1abfed9ef9744274c1e66c6ac12f6a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2caf2490dde9e6bca18f861979d838c9b1abfed9ef9744274c1e66c6ac12f6a6/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3de02f77db8a82d890509743cb0efee95efbd63a22919932eedb4237664ef034/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3de02f77db8a82d890509743cb0efee95efbd63a22919932eedb4237664ef034/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6253727e873eed1a0617db25d3a223df4c0f4f0a6cefc5779e0f3a24dc419ae8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6253727e873eed1a0617db25d3a223df4c0f4f0a6cefc5779e0f3a24dc419ae8/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6566e567d3d183707f7ee5753ca91345b65ba87361543afefdc82f0cf210d654/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6566e567d3d183707f7ee5753ca91345b65ba87361543afefdc82f0cf210d654/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/798cd14fdb96c4e8c773936493eb9d83844d0cdbc4632928f5cd5c60b94a3d8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/798cd14fdb96c4e8c773936493eb9d83844d0cdbc4632928f5cd5c60b94a3d8c/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d7573a0ee248b4183fbc5ae615a5eadc929fb47464dc58c3b5f5a30dae5958c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d7573a0ee248b4183fbc5ae615a5eadc929fb47464dc58c3b5f5a30dae5958c/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ea3da950d87b1423934d69c0b2b95bf7057f33aaa382e1d200f8269229b9c0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ea3da950d87b1423934d69c0b2b95bf7057f33aaa382e1d200f8269229b9c0a/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/926623d0c661fe33616eaeca447149441d5d295dc2dd90534c80533adb588b21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/926623d0c661fe33616eaeca447149441d5d295dc2dd90534c80533adb588b21/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d8e93a4469889ffb7b33be3d5838737c48d16ae2325ab5acb425a6cbf1ac314/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d8e93a4469889ffb7b33be3d5838737c48d16ae2325ab5acb425a6cbf1ac314/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9fc7f744d19c4b75767b7635c4bfdb672ebd771bdfb2eaedce7d3f5d30151e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9fc7f744d19c4b75767b7635c4bfdb672ebd771bdfb2eaedce7d3f5d30151e7/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/beb5e664fd0e98dc0a7e5256f71f5c9df12431436100b8e0f3cc165d097264ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/beb5e664fd0e98dc0a7e5256f71f5c9df12431436100b8e0f3cc165d097264ec/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8a40a08560c8a5aa382406d83ac44aa536b1c02f40f06f8b1dd91dfc4ab6ce0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8a40a08560c8a5aa382406d83ac44aa536b1c02f40f06f8b1dd91dfc4ab6ce0/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c941c1c439823ba7453351ffba09a6ef0a3f801cfec417e7f7f5bdc7a639de3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c941c1c439823ba7453351ffba09a6ef0a3f801cfec417e7f7f5bdc7a639de3e/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d94adc8d6a13bba4e7e21f9fb208b17d97d0b305c8538e3fce7749331c5c1a3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d94adc8d6a13bba4e7e21f9fb208b17d97d0b305c8538e3fce7749331c5c1a3e/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e76075a4085499ba41418da6dfa8bc4eab47352b6980ff99e56a7579ebce5d32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e76075a4085499ba41418da6dfa8bc4eab47352b6980ff99e56a7579ebce5d32/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edfcc54666535d456059b2c9f25f78e0a3ab73ae0461ba5d9edd0809589b7c08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edfcc54666535d456059b2c9f25f78e0a3ab73ae0461ba5d9edd0809589b7c08/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06d9b61a-283a-4ff0-b132-9319897c7e9b/volumes/kubernetes.io~projected/kube-api-access-tzwqg:{mountpoint:/var/lib/kubelet/pods/06d9b61a-283a-4ff0-b132-9319897c7e9b/volumes/kubernetes.io~projected/kube-api-access-tzwqg major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bb8ae2c-9126-4e64-b930-c748b4a44bf1/volumes/kubernetes.io~projected/kube-api-access-5xs7r:{mountpoint:/var/lib/kubelet/pods/0bb8ae2c-9126-4e64-b930-c748b4a44bf1/volumes/kubernetes.io~projected/kube-api-access-5xs7r major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b0926a1-74b6-4510-8fbc-91609d959cbf/volumes/kubernetes.io~projected/kube-api-access-srm2k:{mountpoint:/var/lib/kubelet/pods/2b0926a1-74b6-4510-8fbc-91609d959cbf/volumes/kubernetes.io~projected/kube-api-access-srm2k major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109/volumes/kubernetes.io~projected/kube-api-access-mvtl7:{mountpoint:/var/lib/kubelet/pods/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109/volumes/kubernetes.io~projected/kube-api-access-mvtl7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volumes/kubernetes.io~projected/kube-api-access-8xfdd:{mountpoint:/var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volumes/kubernetes.io~projected/kube-api-access-8xfdd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~projected/kube-api-access-7wtd6:{mountpoint:/var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~projected/kube-api-access-7wtd6 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc/volumes/kubernetes.io~projected/kube-api-access-78tfv:{mountpoint:/var/lib/kubelet/pods/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc/volumes/kubernetes.io~projected/kube-api-access-78tfv major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3084f525-41f6-44e7-9cf4-e0a85e23362e/volumes/kubernetes.io~projected/kube-api-access-k2hxx:{mountpoint:/var/lib/kubelet/pods/3084f525-41f6-44e7-9cf4-e0a85e23362e/volumes/kubernetes.io~projected/kube-api-access-k2hxx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31fce4ed-e652-4919-9cae-157650ca980f/volumes/kubernetes.io~projected/kube-api-access-974nn:{mountpoint:/var/lib/kubelet/pods/31fce4ed-e652-4919-9cae-157650ca980f/volumes/kubernetes.io~projected/kube-api-access-974nn major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0/volumes/kubernetes.io~projected/kube-api-access-snpkt:{mountpoint:/var/lib/kubelet/pods/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0/volumes/kubernetes.io~projected/kube-api-access-snpkt major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61c90821-34fa-4790-a80c-34e6fb6e02ce/volumes/kubernetes.io~projected/kube-api-access-wwvph:{mountpoint:/var/lib/kubelet/pods/61c90821-34fa-4790-a80c-34e6fb6e02ce/volumes/kubernetes.io~projected/kube-api-access-wwvph major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61c90821-34fa-4790-a80c-34e6fb6e02ce/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/61c90821-34fa-4790-a80c-34e6fb6e02ce/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62ad0a1a-eede-46f0-8e64-88b9d86a5239/volumes/kubernetes.io~projected/kube-api-access-ksnbn:{mountpoint:/var/lib/kubelet/pods/62ad0a1a-eede-46f0-8e64-88b9d86a5239/volumes/kubernetes.io~projected/kube-api-access-ksnbn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~projected/kube-api-access-5sc4x:{mountpoint:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~projected/kube-api-access-5sc4x major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~projected/kube-api-access-t6j59:{mountpoint:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~projected/kube-api-access-t6j59 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70698df1-1b21-49a3-9cb1-131b2b113024/volumes/kubernetes.io~projected/kube-api-access-5jkpt:{mountpoint:/var/lib/kubelet/pods/70698df1-1b21-49a3-9cb1-131b2b113024/volumes/kubernetes.io~projected/kube-api-access-5jkpt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70698df1-1b21-49a3-9cb1-131b2b113024/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/70698df1-1b21-49a3-9cb1-131b2b113024/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b7f4d68-a578-434f-894a-6cf2fbc63cda/volumes/kubernetes.io~projected/kube-api-access-bh4kb:{mountpoint:/var/lib/kubelet/pods/7b7f4d68-a578-434f-894a-6cf2fbc63cda/volumes/kubernetes.io~projected/kube-api-access-bh4kb major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b7f4d68-a578-434f-894a-6cf2fbc63cda/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7b7f4d68-a578-434f-894a-6cf2fbc63cda/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~projected/kube-api-access-5rwlr:{mountpoint:/var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~projected/kube-api-access-5rwlr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/877333f1-3281-4c9a-84be-ac1de10369e4/volumes/kubernetes.io~projected/kube-api-access-dxbjk:{mountpoint:/var/lib/kubelet/pods/877333f1-3281-4c9a-84be-ac1de10369e4/volumes/kubernetes.io~projected/kube-api-access-dxbjk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96f587e0-8984-434f-b1a5-bae1b56c343b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/96f587e0-8984-434f-b1a5-bae1b56c343b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4840983-85d4-48ae-8de5-20aa5de9f42c/volumes/kubernetes.io~projected/kube-api-access-ww4t9:{mountpoint:/var/lib/kubelet/pods/a4840983-85d4-48ae-8de5-20aa5de9f42c/volumes/kubernetes.io~projected/kube-api-access-ww4t9 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4840983-85d4-48ae-8de5-20aa5de9f42c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a4840983-85d4-48ae-8de5-20aa5de9f42c/volumes/kubernetes.io~secret/cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b85f4408-b319-46f6-aef0-87b5efd91eec/volumes/kubernetes.io~projected/kube-api-access-jfntl:{mountpoint:/var/lib/kubelet/pods/b85f4408-b319-46f6-aef0-87b5efd91eec/volumes/kubernetes.io~projected/kube-api-access-jfntl major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93cda06-ee44-4b46-8a1b-1cee671fa4e7/volumes/kubernetes.io~projected/kube-api-access-kpl42:{mountpoint:/var/lib/kubelet/pods/b93cda06-ee44-4b46-8a1b-1cee671fa4e7/volumes/kubernetes.io~projected/kube-api-access-kpl42 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/volumes/kubernetes.io~secret/agent-certs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2d3fa2f-6503-48f1-a830-375faa8c3a38/volumes/kubernetes.io~projected/kube-api-access-jcxk6:{mountpoint:/var/lib/kubelet/pods/c2d3fa2f-6503-48f1-a830-375faa8c3a38/volumes/kubernetes.io~projected/kube-api-access-jcxk6 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6d32d84-47e1-4b90-83e1-3509cd867f17/volumes/kubernetes.io~projected/kube-api-access-7s9z9:{mountpoint:/var/lib/kubelet/pods/c6d32d84-47e1-4b90-83e1-3509cd867f17/volumes/kubernetes.io~projected/kube-api-access-7s9z9 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9906391-b55c-4c9c-bb13-58b6339d1f76/volumes/kubernetes.io~projected/kube-api-access-h2md7:{mountpoint:/var/lib/kubelet/pods/c9906391-b55c-4c9c-bb13-58b6339d1f76/volumes/kubernetes.io~projected/kube-api-access-h2md7 major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceacc798-3509-453a-b3ad-48abc179967b/volumes/kubernetes.io~projected/kube-api-access-7wcbv:{mountpoint:/var/lib/kubelet/pods/ceacc798-3509-453a-b3ad-48abc179967b/volumes/kubernetes.io~projected/kube-api-access-7wcbv major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceacc798-3509-453a-b3ad-48abc179967b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ceacc798-3509-453a-b3ad-48abc179967b/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f/volumes/kubernetes.io~projected/kube-api-access-tpdvv:{mountpoint:/var/lib/kubelet/pods/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f/volumes/kubernetes.io~projected/kube-api-access-tpdvv major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d/volumes/kubernetes.io~projected/kube-api-access-zb9dm:{mountpoint:/var/lib/kubelet/pods/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d/volumes/kubernetes.io~projected/kube-api-access-zb9dm major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6244e4c-72f8-40ce-9a76-2afc5467cebe/volumes/kubernetes.io~projected/kube-api-access-hvfmw:{mountpoint:/var/lib/kubelet/pods/f6244e4c-72f8-40ce-9a76-2afc5467cebe/volumes/kubernetes.io~projected/kube-api-access-hvfmw major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8932125-9eb4-4f7f-b577-c232a34a0520/volumes/kubernetes.io~projected/kube-api-access-fxkds:{mountpoint:/var/lib/kubelet/pods/f8932125-9eb4-4f7f-b577-c232a34a0520/volumes/kubernetes.io~projected/kube-api-access-fxkds major:0 minor:224 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/165e374aa7e612c82fa0e44f0e033f045d846b44af94af3c0cdab2283119d0bf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c749f197e6e46b15a8136629e39d3c4e3a5b1c67c812ffe436057f80c0aa91f2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/86c1dfa4424f7aad0459945eec2383c5f49389b3c643a4e91f67a4447afd35cc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/5a25c801121aa0f477df449e5c4b14a3e6c8e3a89764b77814a88e27235cf013/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/3c678b9a7a2acb48a5829bc777ef1aa7d6f57e524b5c2f42d227db4eea461db2/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a6619d6b9fdaf188cbb23097b96afb88e62147980b9d24e41d2d237bbbcc02ad/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/847ddfc3b6f87169181f6dd08ee5761a85fd61c30f4d5cfa463c5431bc7afe9d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/072becc912a0072b0414f8ceb8cebb9d7f6221eac602bf17578b2557a5b254ed/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/19f940fba8fbf6939e5857acd26e33bea5faeca87b6091f244db7cc19b461415/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d368b11349bf5450cf8711b7dcf06d87e9a3035df81347ab4a7ea36e94d4d160/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2993e295c358afa6299ebbd76f7361928d46c0e0e1757789adc7de84aa5e8551/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/5be668ba61bfbf65fdf1520d722c6f6a59f1421eb6541c281452a3567658fbca/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/de3939a718b2db1ea7a5a73d37acd6716e6ce1b317b865e15438faf5354014f1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/be36aedd703c95d1817e059f5cea3326ca05737b9584b506c0f4d92e0ac7cf4b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c30c380c111daf4dc8edd3f3d288c1a2040c46eca5e82a6d20f6f3b903aab626/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/55e53854b99da58d4e14200e5650e7700c7d25ed3e5e867b12ff10c84d286aca/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/52bdc85825f25c343d0f2805a04873b1b59d76390ac9f37b67ef23145c1a086a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d88abd6bd6a1b95b093515e8b793d2e7e580d042ed3c4264b1d5fe56cdbbe88b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7f15bf5ca7f712e20e9f7f3cbecd15ff9b5d369694454f6b820da3c3aa9aea16/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d8b5515bb4c817e55dbb9f8dc8d7a5b79535bcd9a05a605f211fe91ca7c1b0fe/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/30d1b23902db413319e6d0c5fff15026e0b96d0a61ba76e10ab0074e195a8cdc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/26ca2d0be3ae8cfde4317d3b78ce71ee0ef991a7c675bfbbbac67653337587e3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7a8c02816a4de90f29c658bf15bc941e5a4f7cf2b00c64151034e6adae4a62cf/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/ae3b341b6b2d1305d06be1d8d3825b150722404a35488d88cbacfef022166d72/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/1b256ee88d376cedeabad12f3a1b4795210c4bd4fd65071a0fd8149a7a6556a5/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d2aa26b22065ecc6888efebdb0289873c83495b7cc3c39b78e166bbcbc4db256/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/5ea75a845e93cf6d8c2a5f0f39699a421d746a8f51760fd3a6c49893467cf6fa/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/552a1a101537c5964e054f69318b0fa0a100b42651eac64a979c2b6184fcbfd6/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/498c593926745b8a7c4172e872997b71c12365ce3d73ffba9f1aecf7ab87d731/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/3e548a2e7a8309df2cdaee674a3813caab1d47cae6d4485fb89b2158023e05cc/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/4bc8cd4d9017a99c4bccd68e4258c8248665e9022fcfc93702e8e83da74747d3/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c9f55016611d00ad899a21307e0aab9470ff2ffd98ff7b054ce239d701c5a484/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/bea5a8f54a872effe8d8ec7801dccc0ddd3b24467aa0329fd260c87287ee78da/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/696687c1da5a13b79e2729ae26c713538930529b74b633fe24e7fe85fc30459b/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/330ebfee0eba96a4d235993b13fc7be03b67d560781b3287d870b04b18d89400/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/0b031d39cc0153ead05457baa250dd4724b9daed045ee43d243393b3cbe295ec/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/d8fdd12fa677b324700c398be23c1e608e0e1bee09410e268d3b72af3826cba3/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e3bf6523c851f5cfa500d0cd80dc39ec4948bab8834a73f5c6dc00ea1610e17e/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/bf511b913ffa7f12b542199f84bcbd5dcf5776a8a86c2a347c7076e5dd9189a1/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/ec20ea7ecb6338438040b7cfc27141c03df6a29ddbff92f2cc1faa10fc644950/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/8b6622a5540040316cd1aac7460093746bd5bf6d5b84ab1ff019456567d76c0d/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/5023b41f464e5b40bd2caeed55a4daf6dd617e80ab816b31ad2b0a4df4c18d5c/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/f89a0ca66f81a0eb592baf16b8d8e62c4a1769dda3802b72ac7401ac52af9076/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/3708e9e7d409dac31ee4a9d2e9da2a97cdc514349e508a6135b3e48d4c211739/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/ba95ee6319e2a11890b2f891daab41deaa78761f90c1f1835ee3c8122d3db3f2/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/43ca659b53af0070444cae9590503994e07f1caea5593d77569dd1d27e053c08/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/490196733ee8471130962476dfa70d2e50bc86658f35ee352d800231124eed7a/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/5b9ada2fad24ae4283b3982c78c74da866b93f07eec08c3ef56164a60617f6e9/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/887529a2605a0b175683d66c9e9cd43da05f95f7c4880af0c7e4c33eec2d2733/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/546e138d347e573eee3809fc42e3a918e034141ffec28870e552a3b08c5b79ee/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4ea1252e3689694ea22dd1e5b973692f1cc53677d5cdc6ed2a01eedc24ecc3ca/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/0d2d6807e950e16dc07a81e1ee1dc50a91585a97484e94d10de056a75dc64473/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f5eb79c2f0481b4722db25bb7375b7fb64ceeaa3ebc53c2ad9d77d262a56d445/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/237f5d3d118ae9c81176c3b869cadcbfb6cc6df0ffbc84bafe439824b0cc75f4/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b9ef0eccf9b93e0251f13da4e7fcebaa2ef6951d68c37c4f4481778af76a0ec1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/41928bf569141fa6bd7074ada02ccbafb783b7c26958c11aa6a872d0330203e9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/60741547d451b04f3549cb1c9799788556b2e776de2a94db3d8b4a145550692a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/685f891dc07eace898443d4ae6882261f67128c9ff20597e2647673ef8fd36ae/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.907877    5002 manager.go:217] Machine: {Timestamp:2026-10-30 20:19:33.907420173 +0000 UTC m=+0.056471154 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103422 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2813f99d7b3267d6368a7c0208cbbb SystemUUID:ec2813f9-9d7b-3267-d636-8a7c0208cbbb BootID:39919e08-7b86-4812-829e-49232f14df14 Filesystems:[{Device:/var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b93cda06-ee44-4b46-8a1b-1cee671fa4e7/volumes/kubernetes.io~projected/kube-api-access-kpl42 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edfcc54666535d456059b2c9f25f78e0a3ab73ae0461ba5d9edd0809589b7c08/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bb8ae2c-9126-4e64-b930-c748b4a44bf1/volumes/kubernetes.io~projected/kube-api-access-5xs7r DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volumes/kubernetes.io~projected/kube-api-access-8xfdd DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/877333f1-3281-4c9a-84be-ac1de10369e4/volumes/kubernetes.io~projected/kube-api-access-dxbjk DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b7f4d68-a578-434f-894a-6cf2fbc63cda/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/798cd14fdb96c4e8c773936493eb9d83844d0cdbc4632928f5cd5c60b94a3d8c/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0/volumes/kubernetes.io~projected/kube-api-access-snpkt DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3084f525-41f6-44e7-9cf4-e0a85e23362e/volumes/kubernetes.io~projected/kube-api-access-k2hxx DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61c90821-34fa-4790-a80c-34e6fb6e02ce/volumes/kubernetes.io~projected/kube-api-access-wwvph DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ceacc798-3509-453a-b3ad-48abc179967b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~projected/kube-api-access-5rwlr DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f6244e4c-72f8-40ce-9a76-2afc5467cebe/volumes/kubernetes.io~projected/kube-api-access-hvfmw DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70698df1-1b21-49a3-9cb1-131b2b113024/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4840983-85d4-48ae-8de5-20aa5de9f42c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/96f587e0-8984-434f-b1a5-bae1b56c343b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/beb5e664fd0e98dc0a7e5256f71f5c9df12431436100b8e0f3cc165d097264ec/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61c90821-34fa-4790-a80c-34e6fb6e02ce/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc/volumes/kubernetes.io~projected/kube-api-access-78tfv DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~projected/kube-api-access-t6j59 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8932125-9eb4-4f7f-b577-c232a34a0520/volumes/kubernetes.io~projected/kube-api-access-fxkds DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24008d4ce6ed626d6dd444051831bb0d08ca2022ef6a95ad4d09186871841c91/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31fce4ed-e652-4919-9cae-157650ca980f/volumes/kubernetes.io~projected/kube-api-access-974nn DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f/volumes/kubernetes.io~projected/kube-api-access-tpdvv DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7de73755-c298-441d-a185-bdaf04d2f7ea/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6566e567d3d183707f7ee5753ca91345b65ba87361543afefdc82f0cf210d654/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d7573a0ee248b4183fbc5ae615a5eadc929fb47464dc58c3b5f5a30dae5958c/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/62ad0a1a-eede-46f0-8e64-88b9d86a5239/volumes/kubernetes.io~projected/kube-api-access-ksnbn DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2d3fa2f-6503-48f1-a830-375faa8c3a38/volumes/kubernetes.io~projected/kube-api-access-jcxk6 DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109/volumes/kubernetes.io~projected/kube-api-access-mvtl7 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8a40a08560c8a5aa382406d83ac44aa536b1c02f40f06f8b1dd91dfc4ab6ce0/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1421c65a896b1a21f81006414f1720bb7e71282e2332f279c1f906aa94e019ea/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d/volumes/kubernetes.io~projected/kube-api-access-zb9dm DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a784f7346dae1241766ef8d652276d345487ca8459a063e9f88b987c8ab8a99/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/104d02079ce754ff33e427702f69874ea14666d2db848ab0cde16e05d78600b2/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9fc7f744d19c4b75767b7635c4bfdb672ebd771bdfb2eaedce7d3f5d30151e7/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b7f4d68-a578-434f-894a-6cf2fbc63cda/volumes/kubernetes.io~projected/kube-api-access-bh4kb DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70698df1-1b21-49a3-9cb1-131b2b113024/volumes/kubernetes.io~projected/kube-api-access-5jkpt DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b85f4408-b319-46f6-aef0-87b5efd91eec/volumes/kubernetes.io~projected/kube-api-access-jfntl DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e76075a4085499ba41418da6dfa8bc4eab47352b6980ff99e56a7579ebce5d32/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d94adc8d6a13bba4e7e21f9fb208b17d97d0b305c8538e3fce7749331c5c1a3e/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2b0926a1-74b6-4510-8fbc-91609d959cbf/volumes/kubernetes.io~projected/kube-api-access-srm2k DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2dfc646e-d42e-44ad-8a5c-09e8115c06ca/volumes/kubernetes.io~projected/kube-api-access-7wtd6 DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ea3da950d87b1423934d69c0b2b95bf7057f33aaa382e1d200f8269229b9c0a/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2caf2490dde9e6bca18f861979d838c9b1abfed9ef9744274c1e66c6ac12f6a6/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2d15d48f-5197-4096-9011-d82bcb96fde9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/146a6b5d870e6775e806055b841d849b94c7248fd43818abe2dde7641442397c/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6253727e873eed1a0617db25d3a223df4c0f4f0a6cefc5779e0f3a24dc419ae8/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c941c1c439823ba7453351ffba09a6ef0a3f801cfec417e7f7f5bdc7a639de3e/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d8e93a4469889ffb7b33be3d5838737c48d16ae2325ab5acb425a6cbf1ac314/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c6d32d84-47e1-4b90-83e1-3509cd867f17/volumes/kubernetes.io~projected/kube-api-access-7s9z9 DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/140e14d58453a7fbe1994328bde86a8e48131cbe3dc77c1ee84ac9a85383f2fb/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ceacc798-3509-453a-b3ad-48abc179967b/volumes/kubernetes.io~projected/kube-api-access-7wcbv DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4840983-85d4-48ae-8de5-20aa5de9f42c/volumes/kubernetes.io~projected/kube-api-access-ww4t9 DeviceMajor:0 DeviceMinor:239 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/06d9b61a-283a-4ff0-b132-9319897c7e9b/volumes/kubernetes.io~projected/kube-api-access-tzwqg DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/926623d0c661fe33616eaeca447149441d5d295dc2dd90534c80533adb588b21/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes/kubernetes.io~projected/kube-api-access-5sc4x DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9906391-b55c-4c9c-bb13-58b6339d1f76/volumes/kubernetes.io~projected/kube-api-access-h2md7 DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3de02f77db8a82d890509743cb0efee95efbd63a22919932eedb4237664ef034/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:104d02079ce754f MacAddress:6a:c6:23:e6:14:13 Speed:10000 Mtu:8901} {Name:140e14d58453a7f MacAddress:9a:c3:7d:21:c9:65 Speed:10000 Mtu:8901} {Name:1421c65a896b1a2 MacAddress:82:9a:06:a0:d7:e8 Speed:10000 Mtu:8901} {Name:146a6b5d870e677 MacAddress:6a:d8:f2:21:73:f3 Speed:10000 Mtu:8901} {Name:24008d4ce6ed626 MacAddress:4e:20:13:72:bf:c9 Speed:10000 Mtu:8901} {Name:2caf2490dde9e6b MacAddress:aa:3e:3c:27:56:9f Speed:10000 Mtu:8901} {Name:6253727e873eed1 MacAddress:9a:bb:a9:bd:b1:3c Speed:10000 Mtu:8901} {Name:798cd14fdb96c4e MacAddress:9e:32:f5:f1:fb:9d Speed:10000 Mtu:8901} {Name:8d7573a0ee248b4 MacAddress:76:f6:1f:22:e8:76 Speed:10000 Mtu:8901} {Name:8ea3da950d87b14 MacAddress:22:de:a0:54:8b:f6 Speed:10000 Mtu:8901} {Name:9d8e93a4469889f MacAddress:d6:56:ff:de:82:3a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3f:bb:60:53:0f Speed:0 Mtu:9001} {Name:br-int MacAddress:de:55:20:80:90:a2 Speed:0 Mtu:8901} {Name:c8a40a08560c8a5 MacAddress:b2:5d:57:4d:44:f1 Speed:10000 Mtu:8901} {Name:d94adc8d6a13bba MacAddress:de:6c:27:59:b2:56 Speed:10000 Mtu:8901} {Name:e76075a4085499b MacAddress:4a:bd:6e:8e:8b:d0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3f:bb:60:53:0f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:82:94:f9:3c:c7:56 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:1d:9c:99:8e:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908248    5002 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908288    5002 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908593    5002 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908614    5002 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-29.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908728    5002 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908735    5002 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908746    5002 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908764    5002 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908782    5002 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.908933    5002 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909039    5002 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909070    5002 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909083    5002 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909092    5002 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909100    5002 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909108    5002 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909861    5002 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.909871    5002 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:33.912040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.910311    5002 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:19:33.913406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913389    5002 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:19:33.913406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913407    5002 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:33.913554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913518    5002 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:19:33.913756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913740    5002 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:19:33.913833 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913781    5002 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:19:33.913905 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913894    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:19:33.913963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913914    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:19:33.913963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913924    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:19:33.913963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913932    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:19:33.913963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913943    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:19:33.913963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913952    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913969    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913979    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913989    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.913998    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.914017    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.914032    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.914066    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:19:33.914196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.914076    5002 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:19:33.917406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.917389    5002 server.go:1263] "Started kubelet"
Oct 30 20:19:33.917490 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.917452    5002 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:19:33.917939 ip-10-0-141-29 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:19:33.918957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.918294    5002 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:19:33.918957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.918365    5002 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:19:33.918957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.918663    5002 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:19:33.922165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.920851    5002 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.922852    5002 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.922873    5002 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.922945    5002 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.922940    5002 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:13:54 +0000 UTC" deadline="2028-08-06 09:14:28.148477242 +0000 UTC"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.922962    5002 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.922963    5002 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15492h54m54.225517576s"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.922952    5002 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:19:33.923523 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:33.922972    5002 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.924982    5002 factory.go:55] Registering systemd factory
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.924999    5002 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.926211    5002 factory.go:153] Registering CRI-O factory
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.926226    5002 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.926277    5002 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.926296    5002 factory.go:103] Registering Raw factory
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.926310    5002 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.926717    5002 manager.go:319] Starting recovery of all containers
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927259    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8932125-9eb4-4f7f-b577-c232a34a0520" volumeName="kubernetes.io/projected/f8932125-9eb4-4f7f-b577-c232a34a0520-kube-api-access-fxkds" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927297    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dfc646e-d42e-44ad-8a5c-09e8115c06ca" volumeName="kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-stats-auth" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927312    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d797780-22b8-4dee-bb0c-e2b0a38cceac" volumeName="kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927326    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b7f4d68-a578-434f-894a-6cf2fbc63cda" volumeName="kubernetes.io/projected/7b7f4d68-a578-434f-894a-6cf2fbc63cda-kube-api-access-bh4kb" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927337    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceacc798-3509-453a-b3ad-48abc179967b" volumeName="kubernetes.io/projected/ceacc798-3509-453a-b3ad-48abc179967b-kube-api-access-7wcbv" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927349    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b0926a1-74b6-4510-8fbc-91609d959cbf" volumeName="kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927360    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d797780-22b8-4dee-bb0c-e2b0a38cceac" volumeName="kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927371    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d797780-22b8-4dee-bb0c-e2b0a38cceac" volumeName="kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927383    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b85f4408-b319-46f6-aef0-87b5efd91eec" volumeName="kubernetes.io/projected/b85f4408-b319-46f6-aef0-87b5efd91eec-kube-api-access-jfntl" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927394    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceacc798-3509-453a-b3ad-48abc179967b" volumeName="kubernetes.io/secret/ceacc798-3509-453a-b3ad-48abc179967b-serving-cert" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927405    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6244e4c-72f8-40ce-9a76-2afc5467cebe" volumeName="kubernetes.io/projected/f6244e4c-72f8-40ce-9a76-2afc5467cebe-kube-api-access-hvfmw" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927415    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b0926a1-74b6-4510-8fbc-91609d959cbf" volumeName="kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927460    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d15d48f-5197-4096-9011-d82bcb96fde9" volumeName="kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927473    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61c90821-34fa-4790-a80c-34e6fb6e02ce" volumeName="kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927487    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66fe23b4-c77d-49af-95de-6e87cc4eb223" volumeName="kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927512    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4840983-85d4-48ae-8de5-20aa5de9f42c" volumeName="kubernetes.io/projected/a4840983-85d4-48ae-8de5-20aa5de9f42c-kube-api-access-ww4t9" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927525    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6d32d84-47e1-4b90-83e1-3509cd867f17" volumeName="kubernetes.io/configmap/c6d32d84-47e1-4b90-83e1-3509cd867f17-config-volume" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927557    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d34f102b-c1f4-4e19-a7ce-cdc1ea51930f" volumeName="kubernetes.io/projected/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f-kube-api-access-tpdvv" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927569    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61c90821-34fa-4790-a80c-34e6fb6e02ce" volumeName="kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-service-ca-bundle" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927581    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66fe23b4-c77d-49af-95de-6e87cc4eb223" volumeName="kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927593    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66fe23b4-c77d-49af-95de-6e87cc4eb223" volumeName="kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927607    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d" volumeName="kubernetes.io/projected/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d-kube-api-access-zb9dm" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927620    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ad0a1a-eede-46f0-8e64-88b9d86a5239" volumeName="kubernetes.io/projected/62ad0a1a-eede-46f0-8e64-88b9d86a5239-kube-api-access-ksnbn" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927633    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70698df1-1b21-49a3-9cb1-131b2b113024" volumeName="kubernetes.io/configmap/70698df1-1b21-49a3-9cb1-131b2b113024-config" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927644    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93cda06-ee44-4b46-8a1b-1cee671fa4e7" volumeName="kubernetes.io/projected/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kube-api-access-kpl42" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927655    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61c90821-34fa-4790-a80c-34e6fb6e02ce" volumeName="kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-tmp" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927667    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ad0a1a-eede-46f0-8e64-88b9d86a5239" volumeName="kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927680    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66fe23b4-c77d-49af-95de-6e87cc4eb223" volumeName="kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927692    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70698df1-1b21-49a3-9cb1-131b2b113024" volumeName="kubernetes.io/secret/70698df1-1b21-49a3-9cb1-131b2b113024-serving-cert" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927700    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b7f4d68-a578-434f-894a-6cf2fbc63cda" volumeName="kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-config" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927707    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de73755-c298-441d-a185-bdaf04d2f7ea" volumeName="kubernetes.io/projected/7de73755-c298-441d-a185-bdaf04d2f7ea-kube-api-access-5rwlr" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927722    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceacc798-3509-453a-b3ad-48abc179967b" volumeName="kubernetes.io/configmap/ceacc798-3509-453a-b3ad-48abc179967b-config" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927730    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f260119-6cc6-44c8-b7e7-c8c6ae3471fc" volumeName="kubernetes.io/configmap/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-config" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927739    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f260119-6cc6-44c8-b7e7-c8c6ae3471fc" volumeName="kubernetes.io/projected/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-kube-api-access-78tfv" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927747    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61c90821-34fa-4790-a80c-34e6fb6e02ce" volumeName="kubernetes.io/secret/61c90821-34fa-4790-a80c-34e6fb6e02ce-serving-cert" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927754    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ad0a1a-eede-46f0-8e64-88b9d86a5239" volumeName="kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927763    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b7f4d68-a578-434f-894a-6cf2fbc63cda" volumeName="kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-trusted-ca" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927771    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877333f1-3281-4c9a-84be-ac1de10369e4" volumeName="kubernetes.io/projected/877333f1-3281-4c9a-84be-ac1de10369e4-kube-api-access-dxbjk" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927778    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bb8ae2c-9126-4e64-b930-c748b4a44bf1" volumeName="kubernetes.io/configmap/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-telemetry-config" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927787    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70698df1-1b21-49a3-9cb1-131b2b113024" volumeName="kubernetes.io/projected/70698df1-1b21-49a3-9cb1-131b2b113024-kube-api-access-5jkpt" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927794    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4840983-85d4-48ae-8de5-20aa5de9f42c" volumeName="kubernetes.io/secret/a4840983-85d4-48ae-8de5-20aa5de9f42c-cert" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927813    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d15d48f-5197-4096-9011-d82bcb96fde9" volumeName="kubernetes.io/secret/2d15d48f-5197-4096-9011-d82bcb96fde9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927820    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61c90821-34fa-4790-a80c-34e6fb6e02ce" volumeName="kubernetes.io/projected/61c90821-34fa-4790-a80c-34e6fb6e02ce-kube-api-access-wwvph" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927827    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ad0a1a-eede-46f0-8e64-88b9d86a5239" volumeName="kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927833    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d797780-22b8-4dee-bb0c-e2b0a38cceac" volumeName="kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927841    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de73755-c298-441d-a185-bdaf04d2f7ea" volumeName="kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-tmp" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927848    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877333f1-3281-4c9a-84be-ac1de10369e4" volumeName="kubernetes.io/configmap/877333f1-3281-4c9a-84be-ac1de10369e4-serviceca" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927855    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dfc646e-d42e-44ad-8a5c-09e8115c06ca" volumeName="kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-default-certificate" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927861    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3084f525-41f6-44e7-9cf4-e0a85e23362e" volumeName="kubernetes.io/projected/3084f525-41f6-44e7-9cf4-e0a85e23362e-kube-api-access-k2hxx" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927869    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f15b82-a100-493b-aef0-c83d07ad1806" volumeName="kubernetes.io/configmap/71f15b82-a100-493b-aef0-c83d07ad1806-nginx-conf" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927877    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f" volumeName="kubernetes.io/secret/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-agent-certs" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927885    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5687ec3f-b05f-4e0b-b4ce-04f50516b5d0" volumeName="kubernetes.io/empty-dir/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-data-volume" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927893    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5687ec3f-b05f-4e0b-b4ce-04f50516b5d0" volumeName="kubernetes.io/projected/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-api-access-snpkt" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927900    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66fe23b4-c77d-49af-95de-6e87cc4eb223" volumeName="kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927907    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66fe23b4-c77d-49af-95de-6e87cc4eb223" volumeName="kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927914    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d797780-22b8-4dee-bb0c-e2b0a38cceac" volumeName="kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927928    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b7f4d68-a578-434f-894a-6cf2fbc63cda" volumeName="kubernetes.io/secret/7b7f4d68-a578-434f-894a-6cf2fbc63cda-serving-cert" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927938    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6d32d84-47e1-4b90-83e1-3509cd867f17" volumeName="kubernetes.io/empty-dir/c6d32d84-47e1-4b90-83e1-3509cd867f17-tmp-dir" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927944    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d003e73-1f2d-4e65-a6dc-f8ead9bc6109" volumeName="kubernetes.io/configmap/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927951    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d15d48f-5197-4096-9011-d82bcb96fde9" volumeName="kubernetes.io/projected/2d15d48f-5197-4096-9011-d82bcb96fde9-kube-api-access-8xfdd" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927958    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dfc646e-d42e-44ad-8a5c-09e8115c06ca" volumeName="kubernetes.io/projected/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-kube-api-access-7wtd6" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927965    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f" volumeName="kubernetes.io/configmap/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927975    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6d32d84-47e1-4b90-83e1-3509cd867f17" volumeName="kubernetes.io/projected/c6d32d84-47e1-4b90-83e1-3509cd867f17-kube-api-access-7s9z9" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927983    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bb8ae2c-9126-4e64-b930-c748b4a44bf1" volumeName="kubernetes.io/projected/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-kube-api-access-5xs7r" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927989    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d15d48f-5197-4096-9011-d82bcb96fde9" volumeName="kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-config" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.927998    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61c90821-34fa-4790-a80c-34e6fb6e02ce" volumeName="kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-snapshots" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928005    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6244e4c-72f8-40ce-9a76-2afc5467cebe" volumeName="kubernetes.io/empty-dir/f6244e4c-72f8-40ce-9a76-2afc5467cebe-tmp-dir" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928012    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b0926a1-74b6-4510-8fbc-91609d959cbf" volumeName="kubernetes.io/projected/2b0926a1-74b6-4510-8fbc-91609d959cbf-kube-api-access-srm2k" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928018    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d003e73-1f2d-4e65-a6dc-f8ead9bc6109" volumeName="kubernetes.io/projected/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-kube-api-access-mvtl7" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928029    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d797780-22b8-4dee-bb0c-e2b0a38cceac" volumeName="kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928040    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de73755-c298-441d-a185-bdaf04d2f7ea" volumeName="kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-tuned" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928048    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d15d48f-5197-4096-9011-d82bcb96fde9" volumeName="kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-env-overrides" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928055    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d797780-22b8-4dee-bb0c-e2b0a38cceac" volumeName="kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928063    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9906391-b55c-4c9c-bb13-58b6339d1f76" volumeName="kubernetes.io/projected/c9906391-b55c-4c9c-bb13-58b6339d1f76-kube-api-access-h2md7" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928070    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06d9b61a-283a-4ff0-b132-9319897c7e9b" volumeName="kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928078    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31fce4ed-e652-4919-9cae-157650ca980f" volumeName="kubernetes.io/projected/31fce4ed-e652-4919-9cae-157650ca980f-kube-api-access-974nn" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928085    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66fe23b4-c77d-49af-95de-6e87cc4eb223" volumeName="kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928092    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96f587e0-8984-434f-b1a5-bae1b56c343b" volumeName="kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928099    5002 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d3fa2f-6503-48f1-a830-375faa8c3a38" volumeName="kubernetes.io/projected/c2d3fa2f-6503-48f1-a830-375faa8c3a38-kube-api-access-jcxk6" seLinuxMountContext=""
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928106    5002 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:19:33.929551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.928111    5002 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:19:33.944798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.944741    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/1.log"
Oct 30 20:19:33.945130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.945114    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/1.log"
Oct 30 20:19:33.956785 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.956724    5002 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:19:33.958151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.958131    5002 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:19:33.958224 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.958159    5002 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:19:33.958297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.958286    5002 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:19:33.958395 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:33.958352    5002 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:19:33.975069 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.975041    5002 generic.go:356] "Generic (PLEG): container finished" podID="2203f4a44f682afa473e5863b4adcd60" containerID="1a7de360c89f4a57884ceefb797302ec4160434d8a82c9fda837951a36274bde" exitCode=0
Oct 30 20:19:33.989286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.989269    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/1.log"
Oct 30 20:19:33.989362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:33.989297    5002 generic.go:356] "Generic (PLEG): container finished" podID="7b7f4d68-a578-434f-894a-6cf2fbc63cda" containerID="6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b" exitCode=255
Oct 30 20:19:34.008098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.008071    5002 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="a77a2717ef98ef53c722fc7570efc4da2e35e020f7e19cb157583ed58f8a40a0" exitCode=0
Oct 30 20:19:34.008098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.008094    5002 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="d78c3ac22c7a6920dc72361db31a2239b4edb8234497d620b3407a7d8fb85312" exitCode=0
Oct 30 20:19:34.008098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.008101    5002 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="87153e04e25c6a9834bc568162fcda58ce4f662566ce73475ff5a756fbb4a39d" exitCode=0
Oct 30 20:19:34.008265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.008107    5002 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="748c1163e5aa38a19e75f1cf26a6ac28e1e5528701142a71bb3a4db6b79e36a0" exitCode=0
Oct 30 20:19:34.008265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.008112    5002 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="cddf0cc2d939edb61be54c9a42299b71c8f16032d0e330c17de659270e8eab2e" exitCode=0
Oct 30 20:19:34.008265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.008117    5002 generic.go:356] "Generic (PLEG): container finished" podID="62ad0a1a-eede-46f0-8e64-88b9d86a5239" containerID="df49543dc330e61bebb644a50c75d65bc561c09207fdae1fe0dda16c23de9c6d" exitCode=0
Oct 30 20:19:34.023083 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.023055    5002 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:19:34.024569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.024554    5002 manager.go:324] Recovery completed
Oct 30 20:19:34.040946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.040930    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:34.043972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.043957    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:34.044035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.043985    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:34.044035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.043995    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:34.046037 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046021    5002 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:19:34.046037 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046034    5002 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:19:34.046194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046038    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:34.046194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046053    5002 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:19:34.046292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046205    5002 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:19:34.046292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046221    5002 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:19:34.046292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046246    5002 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:19:34.046292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046254    5002 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:19:34.046292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046262    5002 policy_none.go:50] "Start"
Oct 30 20:19:34.046292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046274    5002 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:19:34.046292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046286    5002 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:19:34.046626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046461    5002 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:19:34.046626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046471    5002 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:19:34.046626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.046478    5002 policy_none.go:44] "Start"
Oct 30 20:19:34.048099 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.048079    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:34.048179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.048105    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:34.048179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.048115    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:34.051597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.051569    5002 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:19:34.054150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054134    5002 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:19:34.054221 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.054158    5002 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:19:34.054221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054167    5002 server.go:87] "Starting device plugin registration server"
Oct 30 20:19:34.054465 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054453    5002 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:19:34.054505 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054467    5002 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:19:34.054576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054558    5002 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:19:34.054711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054673    5002 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:19:34.054806 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054717    5002 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:19:34.054806 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054724    5002 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:19:34.054884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.054765    5002 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:19:34.054707039Z","UUID":"ca09afc3-8220-4619-8484-1f3482e16591","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:19:34.056109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.056092    5002 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:19:34.056189 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.056116    5002 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:19:34.056239 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.056204    5002 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:19:34.056289 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.056241    5002 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-29.us-west-1.compute.internal\" not found"
Oct 30 20:19:34.058472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.058437    5002 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"]
Oct 30 20:19:34.058703 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.058683    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:34.059551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.059514    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:34.059630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.059563    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:34.059630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.059592    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:34.059897 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.059884    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:34.060059 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060042    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:34.060674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060659    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:34.060773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060680    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:34.060773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060691    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:34.060773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060712    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:34.060773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060736    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:34.060773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060751    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:34.061011 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.060910    5002 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.061011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060932    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"a97eddfe748e1e815e76819b9d90360d","Type":"ContainerStarted","Data":"69697f617293cf91c6a227759524249cd4e4b4a42f46d6ce50a0f546bae71312"}
Oct 30 20:19:34.061011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060962    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"a97eddfe748e1e815e76819b9d90360d","Type":"ContainerStarted","Data":"1cb05e1d53477bc0aeb9672b34f193e6d5735c15f66d7ce8f9f28cd7f01db6ac"}
Oct 30 20:19:34.061011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060974    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"2203f4a44f682afa473e5863b4adcd60","Type":"ContainerStarted","Data":"f36e46557a88cbe66299f6ff4f8051f4ff22470b719723ce2a3a96cce2f9b846"}
Oct 30 20:19:34.061011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.060985    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"2203f4a44f682afa473e5863b4adcd60","Type":"ContainerDied","Data":"1a7de360c89f4a57884ceefb797302ec4160434d8a82c9fda837951a36274bde"}
Oct 30 20:19:34.061011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.061002    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal" event={"ID":"2203f4a44f682afa473e5863b4adcd60","Type":"ContainerStarted","Data":"9603960dd72fe185f70d26cfadf19e9aff7cb5dcc89fbcd333fd26e0af077d11"}
Oct 30 20:19:34.061260 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.061134    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:34.061889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.061875    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:34.061968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.061898    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:34.061968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.061913    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:34.062051 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.062028    5002 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.us-west-1.compute.internal\" not found" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.120248 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.120226    5002 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:34.120248 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.120242    5002 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:34.120386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.120255    5002 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:19:34.120386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.120265    5002 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:34.123593 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.123577    5002 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.154688 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.154658    5002 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:34.155424 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.155408    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:34.155502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.155435    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:34.155502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.155449    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:34.155502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.155491    5002 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.171098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.171079    5002 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:34.185441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.185422    5002 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:34.185515 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.185466    5002 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.185515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.185481    5002 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.220912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.220888    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.220982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.220916    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.220982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.220934    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a97eddfe748e1e815e76819b9d90360d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"a97eddfe748e1e815e76819b9d90360d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.272356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.272337    5002 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.272426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.272384    5002 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.272794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.272781    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 20:19:34.272851 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.272806    5002 setters.go:546] "Node became not ready" node="ip-10-0-141-29.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T20:19:34Z","lastTransitionTime":"2026-10-30T20:19:34Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 20:19:34.283685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.283666    5002 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-29.us-west-1.compute.internal" is owned by different node (oldNodeID="6637dd13-0348-4d2e-b60a-e8ee34765c1e", newNodeID=""), cleaning up...
Oct 30 20:19:34.284070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.284057    5002 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:34.284118 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:34.284086    5002 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.321237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.321212    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.321332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.321281    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.321332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.321297    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.321332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.321327    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a97eddfe748e1e815e76819b9d90360d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"a97eddfe748e1e815e76819b9d90360d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.321442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.321368    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a97eddfe748e1e815e76819b9d90360d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"a97eddfe748e1e815e76819b9d90360d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.321442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.321380    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203f4a44f682afa473e5863b4adcd60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" (UID: \"2203f4a44f682afa473e5863b4adcd60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:34.910062 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:34.910031    5002 apiserver.go:52] "Watching apiserver"
Oct 30 20:19:35.015783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.015762    5002 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:35.066126 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.066104    5002 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:19:35.066936 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.066904    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7","openshift-multus/multus-g6df8","kube-system/kas-connection-checker-685895c7c9-dqz7k","kube-system/kas-connection-checker-685895c7c9-df452","openshift-network-console/networking-console-plugin-86c478bb88-hdjsr","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676","openshift-image-registry/image-registry-695dcdfbc7-z5fr7","openshift-dns/node-resolver-b5kfh","openshift-dns/dns-default-8hmbp","openshift-ovn-kubernetes/ovnkube-node-6vhjk","openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn","openshift-multus/multus-additional-cni-plugins-r25dm","openshift-network-operator/iptables-alerter-nv9fr","kube-system/konnectivity-agent-l4ldp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz","openshift-console-operator/console-operator-7cd56b8dbc-jnzz2","openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw","openshift-insights/insights-operator-7c4986d449-mjg55","openshift-insights/insights-runtime-extractor-cmnm9","openshift-multus/network-metrics-daemon-svcz9","kube-system/kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-n6f9w","openshift-cluster-node-tuning-operator/tuned-zz2fw","kube-system/kas-connection-checker-685895c7c9-nr6ns","openshift-image-registry/node-ca-x2lgj","openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5","openshift-ingress/router-default-67cdd9f98c-zf64k","openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67","openshift-image-registry/image-registry-7f4544dfb8-94842","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-nnc2v","kube-system/global-pull-secret-syncer-jkspb"]
Oct 30 20:19:35.068178 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.068151    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.068495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.068468    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.069118 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.069095    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.069471 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.069452    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.072068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.072052    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:35.075322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.075303    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:35.077642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.077609    5002 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:35.077759 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.077685    5002 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.us-west-1.compute.internal"
Oct 30 20:19:35.081366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.081346    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.081827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.081815    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.083427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.083407    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.122145 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122121    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.122236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122174    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 20:19:35.122236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122217    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.122337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122237    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.122387 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122355    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.122387 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122357    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.122475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122387    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.122475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122388    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.122475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122422    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.122646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122629    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 20:19:35.122787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122774    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 20:19:35.122827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122814    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4tp44\""
Oct 30 20:19:35.122866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.122853    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6vlmd\""
Oct 30 20:19:35.123205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123189    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 20:19:35.123294 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123262    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:19:35.123393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123323    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.123502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123483    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 20:19:35.123597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123555    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:19:35.123663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123601    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.123663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123607    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.123756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123611    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 20:19:35.123756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123676    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.123756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123643    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.123756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.123619    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 20:19:35.124035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124008    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.124340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124323    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 20:19:35.124431 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124404    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 20:19:35.124488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124431    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 20:19:35.124488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124448    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:19:35.124682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124627    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 20:19:35.124784 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124759    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hbnvb\""
Oct 30 20:19:35.124915 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124897    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q5fsc\""
Oct 30 20:19:35.124915 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124911    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ckn8f\""
Oct 30 20:19:35.125018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.124945    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fb4b4\""
Oct 30 20:19:35.125402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.125383    5002 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:19:35.128151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128134    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.128228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128161    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:19:35.128228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128161    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-daemon-config\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.128228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128202    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-kubelet\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.128228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128219    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-bin\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.128390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128240    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxk6\" (UniqueName: \"kubernetes.io/projected/c2d3fa2f-6503-48f1-a830-375faa8c3a38-kube-api-access-jcxk6\") pod \"kas-connection-checker-685895c7c9-nr6ns\" (UID: \"c2d3fa2f-6503-48f1-a830-375faa8c3a38\") " pod="kube-system/kas-connection-checker-685895c7c9-nr6ns"
Oct 30 20:19:35.128390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128266    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwlr\" (UniqueName: \"kubernetes.io/projected/7de73755-c298-441d-a185-bdaf04d2f7ea-kube-api-access-5rwlr\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.128390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128321    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xs7r\" (UniqueName: \"kubernetes.io/projected/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-kube-api-access-5xs7r\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.128515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128449    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hxx\" (UniqueName: \"kubernetes.io/projected/3084f525-41f6-44e7-9cf4-e0a85e23362e-kube-api-access-k2hxx\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:35.128515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128485    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-socket-dir-parent\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.128609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128512    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-systemd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.128653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128639    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-etc-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.128685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128661    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-env-overrides\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.128685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128679    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7f4d68-a578-434f-894a-6cf2fbc63cda-serving-cert\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.128746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128711    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-konnectivity-ca\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:19:35.128793 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128781    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.128859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128797    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-trusted-ca\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.128859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128813    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70698df1-1b21-49a3-9cb1-131b2b113024-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:35.128859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128828    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-var-lib-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.128859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128855    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71f15b82-a100-493b-aef0-c83d07ad1806-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128873    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974nn\" (UniqueName: \"kubernetes.io/projected/31fce4ed-e652-4919-9cae-157650ca980f-kube-api-access-974nn\") pod \"kas-connection-checker-685895c7c9-df452\" (UID: \"31fce4ed-e652-4919-9cae-157650ca980f\") " pod="kube-system/kas-connection-checker-685895c7c9-df452"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128908    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128923    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-node-log\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-data-volume\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128951    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129008    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-data-volume\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.128993    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnbn\" (UniqueName: \"kubernetes.io/projected/62ad0a1a-eede-46f0-8e64-88b9d86a5239-kube-api-access-ksnbn\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.129050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129014    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70698df1-1b21-49a3-9cb1-131b2b113024-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:35.129336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129062    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.129336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129124    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-netns\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.129336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129165    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.129336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129187    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.129336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129249    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.129336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129283    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-snapshots\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.129523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129341    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-snapshots\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.129523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129418    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-netns\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.129523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129441    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.129523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129442    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-bin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.129523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129468    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-kubelet\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.129523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129491    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.129523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129515    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-iptables-alerter-script\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129556    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ceacc798-3509-453a-b3ad-48abc179967b-config\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129573    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129589    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-systemd-units\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129605    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129619    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-dbus\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129672    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d32d84-47e1-4b90-83e1-3509cd867f17-tmp-dir\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129715    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129819    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-ovn\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129819    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.129846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129844    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d32d84-47e1-4b90-83e1-3509cd867f17-tmp-dir\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129860    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpkt\" (UniqueName: \"kubernetes.io/projected/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-api-access-snpkt\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129879    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6j59\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129895    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcbv\" (UniqueName: \"kubernetes.io/projected/ceacc798-3509-453a-b3ad-48abc179967b-kube-api-access-7wcbv\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129911    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-cni-binary-copy\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129927    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-multus\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129942    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.129977    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-script-lib\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130007    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-crio-socket\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130033    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkpt\" (UniqueName: \"kubernetes.io/projected/70698df1-1b21-49a3-9cb1-131b2b113024-kube-api-access-5jkpt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130052    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ceacc798-3509-453a-b3ad-48abc179967b-serving-cert\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130086    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130107    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tfv\" (UniqueName: \"kubernetes.io/projected/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-kube-api-access-78tfv\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130122    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130161    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-etc-kubernetes\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130191    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9z9\" (UniqueName: \"kubernetes.io/projected/c6d32d84-47e1-4b90-83e1-3509cd867f17-kube-api-access-7s9z9\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130217    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130221    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ceacc798-3509-453a-b3ad-48abc179967b-serving-cert\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130241    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.130299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130264    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130323    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfdd\" (UniqueName: \"kubernetes.io/projected/2d15d48f-5197-4096-9011-d82bcb96fde9-kube-api-access-8xfdd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130341    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130348    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130351    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh4kb\" (UniqueName: \"kubernetes.io/projected/7b7f4d68-a578-434f-894a-6cf2fbc63cda-kube-api-access-bh4kb\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130417    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130430    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-host\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130476    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-lib-modules\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130505    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d32d84-47e1-4b90-83e1-3509cd867f17-config-volume\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130550    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-host-slash\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130592    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4t9\" (UniqueName: \"kubernetes.io/projected/a4840983-85d4-48ae-8de5-20aa5de9f42c-kube-api-access-ww4t9\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130625    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6244e4c-72f8-40ce-9a76-2afc5467cebe-tmp-dir\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130644    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-service-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130661    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvph\" (UniqueName: \"kubernetes.io/projected/61c90821-34fa-4790-a80c-34e6fb6e02ce-kube-api-access-wwvph\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130734    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6244e4c-72f8-40ce-9a76-2afc5467cebe-tmp-dir\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130758    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130774    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-sys-fs\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130792    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-system-cni-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130867    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-conf\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130885    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-var-lib-kubelet\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.130922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130915    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkds\" (UniqueName: \"kubernetes.io/projected/f8932125-9eb4-4f7f-b577-c232a34a0520-kube-api-access-fxkds\") pod \"network-check-source-6f78d59f99-fbzx6\" (UID: \"f8932125-9eb4-4f7f-b577-c232a34a0520\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130937    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-kubernetes\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130955    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.130997    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sc4x\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131025    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-kubelet-config\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131036    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131053    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131078    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfntl\" (UniqueName: \"kubernetes.io/projected/b85f4408-b319-46f6-aef0-87b5efd91eec-kube-api-access-jfntl\") pod \"kas-connection-checker-685895c7c9-dqz7k\" (UID: \"b85f4408-b319-46f6-aef0-87b5efd91eec\") " pod="kube-system/kas-connection-checker-685895c7c9-dqz7k"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131096    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfmw\" (UniqueName: \"kubernetes.io/projected/f6244e4c-72f8-40ce-9a76-2afc5467cebe-kube-api-access-hvfmw\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131112    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-tuned\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131134    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131152    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpl42\" (UniqueName: \"kubernetes.io/projected/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kube-api-access-kpl42\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131168    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-default-certificate\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131190    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131233    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131249    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-tuned\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131285    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-run\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131377    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-cnibin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131440    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-device-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131462    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cnibin\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131467    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-default-certificate\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131482    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/2b0926a1-74b6-4510-8fbc-91609d959cbf-kube-api-access-srm2k\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.131506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131508    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131560    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131584    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-stats-auth\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131600    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4840983-85d4-48ae-8de5-20aa5de9f42c-cert\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131625    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-config\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131668    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131755    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d15d48f-5197-4096-9011-d82bcb96fde9-ovn-node-metrics-cert\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131789    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2md7\" (UniqueName: \"kubernetes.io/projected/c9906391-b55c-4c9c-bb13-58b6339d1f76-kube-api-access-h2md7\") pod \"migrator-66df5d496b-d6p67\" (UID: \"c9906391-b55c-4c9c-bb13-58b6339d1f76\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131822    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtd6\" (UniqueName: \"kubernetes.io/projected/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-kube-api-access-7wtd6\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131839    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-modprobe-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131858    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-sys\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131878    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-stats-auth\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131883    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70698df1-1b21-49a3-9cb1-131b2b113024-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131917    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131945    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.131991    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-netd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132027    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132060    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132063    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70698df1-1b21-49a3-9cb1-131b2b113024-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132099    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-hostroot\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132130    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9dm\" (UniqueName: \"kubernetes.io/projected/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d-kube-api-access-zb9dm\") pod \"volume-data-source-validator-58c9f5f46d-x2hmz\" (UID: \"ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132151    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-registration-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132171    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.132302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132225    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/877333f1-3281-4c9a-84be-ac1de10369e4-serviceca\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132322    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-systemd\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132333    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132369    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdvv\" (UniqueName: \"kubernetes.io/projected/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f-kube-api-access-tpdvv\") pod \"migrator-66df5d496b-qlgnw\" (UID: \"d34f102b-c1f4-4e19-a7ce-cdc1ea51930f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132387    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132418    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132445    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6244e4c-72f8-40ce-9a76-2afc5467cebe-hosts-file\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132514    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbjk\" (UniqueName: \"kubernetes.io/projected/877333f1-3281-4c9a-84be-ac1de10369e4-kube-api-access-dxbjk\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132580    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-agent-certs\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132607    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/877333f1-3281-4c9a-84be-ac1de10369e4-serviceca\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132649    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132688    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-tmp\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132724    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7de73755-c298-441d-a185-bdaf04d2f7ea-tmp\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132753    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132781    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132810    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-conf-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132836    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-config\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132873    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132911    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-tmp\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-binary-copy\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.132954    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-system-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133028    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-k8s-cni-cncf-io\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133043    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c90821-34fa-4790-a80c-34e6fb6e02ce-tmp\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133064    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-slash\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133095    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-log-socket\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133112    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61c90821-34fa-4790-a80c-34e6fb6e02ce-serving-cert\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133135    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysconfig\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133175    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-multus-certs\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.133198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133201    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-socket-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.134047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133227    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.134047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133251    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/877333f1-3281-4c9a-84be-ac1de10369e4-host\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:19:35.134047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133292    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtl7\" (UniqueName: \"kubernetes.io/projected/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-kube-api-access-mvtl7\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:19:35.134047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.133297    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.171914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.171867    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:19:35.172660 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.172639    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.172916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.172895    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.184647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.184622    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974nn\" (UniqueName: \"kubernetes.io/projected/31fce4ed-e652-4919-9cae-157650ca980f-kube-api-access-974nn\") pod \"kas-connection-checker-685895c7c9-df452\" (UID: \"31fce4ed-e652-4919-9cae-157650ca980f\") " pod="kube-system/kas-connection-checker-685895c7c9-df452"
Oct 30 20:19:35.184719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.184683    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xs7r\" (UniqueName: \"kubernetes.io/projected/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-kube-api-access-5xs7r\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.185112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.185056    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxk6\" (UniqueName: \"kubernetes.io/projected/c2d3fa2f-6503-48f1-a830-375faa8c3a38-kube-api-access-jcxk6\") pod \"kas-connection-checker-685895c7c9-nr6ns\" (UID: \"c2d3fa2f-6503-48f1-a830-375faa8c3a38\") " pod="kube-system/kas-connection-checker-685895c7c9-nr6ns"
Oct 30 20:19:35.189518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.189502    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkpt\" (UniqueName: \"kubernetes.io/projected/70698df1-1b21-49a3-9cb1-131b2b113024-kube-api-access-5jkpt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7\" (UID: \"70698df1-1b21-49a3-9cb1-131b2b113024\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7"
Oct 30 20:19:35.189704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.189672    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.189863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.189848    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6j59\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.190775 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.190756    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.190973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.190956    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sc4x\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.191894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.191879    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfntl\" (UniqueName: \"kubernetes.io/projected/b85f4408-b319-46f6-aef0-87b5efd91eec-kube-api-access-jfntl\") pod \"kas-connection-checker-685895c7c9-dqz7k\" (UID: \"b85f4408-b319-46f6-aef0-87b5efd91eec\") " pod="kube-system/kas-connection-checker-685895c7c9-dqz7k"
Oct 30 20:19:35.193587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.193571    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtd6\" (UniqueName: \"kubernetes.io/projected/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-kube-api-access-7wtd6\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.193780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.193765    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tfv\" (UniqueName: \"kubernetes.io/projected/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-kube-api-access-78tfv\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.221270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221250    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.221331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221284    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbjk\" (UniqueName: \"kubernetes.io/projected/877333f1-3281-4c9a-84be-ac1de10369e4-kube-api-access-dxbjk\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:19:35.221692 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221652    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.221851 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221789    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.221928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221904    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.222004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221931    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:19:35.222004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221941    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.222004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221954    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.222004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.221934    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.222349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.222278    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.222349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.222294    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.222585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.222478    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.222585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.222488    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:19:35.222585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.222491    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:19:35.223479 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.223361    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9dm\" (UniqueName: \"kubernetes.io/projected/ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d-kube-api-access-zb9dm\") pod \"volume-data-source-validator-58c9f5f46d-x2hmz\" (UID: \"ee7a213a-2533-4cfa-a0a8-2ce6a1bda92d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x2hmz"
Oct 30 20:19:35.223681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.223520    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.223681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.223562    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.223681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.223649    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.224568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.223703    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.224568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224027    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:19:35.224568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224065    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.224568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224088    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.224568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224069    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:19:35.224568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224222    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.224568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224268    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 20:19:35.225220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224852    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-binary-copy\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.225220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224863    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 20:19:35.225220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.224909    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.225220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.225173    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:19:35.225389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.225288    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.225923    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h8r9r\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.225940    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226028    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226053    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4ct2\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226067    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwqg\" (UniqueName: \"kubernetes.io/projected/06d9b61a-283a-4ff0-b132-9319897c7e9b-kube-api-access-tzwqg\") pod \"network-check-target-n6f9w\" (UID: \"06d9b61a-283a-4ff0-b132-9319897c7e9b\") " pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226086    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226192    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226202    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lm5vf\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226249    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vn2fz\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226273    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226320    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226393    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nnssb\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226411    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226399    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226447    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226501    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k74qc\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226621    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vdqtm\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226706    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5x4ch\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226714    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdvv\" (UniqueName: \"kubernetes.io/projected/d34f102b-c1f4-4e19-a7ce-cdc1ea51930f-kube-api-access-tpdvv\") pod \"migrator-66df5d496b-qlgnw\" (UID: \"d34f102b-c1f4-4e19-a7ce-cdc1ea51930f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlgnw"
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226763    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q6sjr\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226806    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226817    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226842    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6wxmv\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226952    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226955    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226981    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.226986    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227047    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227198    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwdx\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227201    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kv88f\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227199    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227204    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wz6nt\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227202    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rnpmh\""
Oct 30 20:19:35.227289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227250    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227370    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gbmn9\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227382    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227439    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l7r4r\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227837    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227854    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwff\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227889    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227895    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227903    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kd6m6\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227934    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227955    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxq5f\""
Oct 30 20:19:35.228693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.227990    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bp5wk\""
Oct 30 20:19:35.229179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.228882    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-daemon-config\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.229179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.228994    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7f4d68-a578-434f-894a-6cf2fbc63cda-serving-cert\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.229179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.229089    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71f15b82-a100-493b-aef0-c83d07ad1806-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:35.229179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.229120    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-env-overrides\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.229288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.229245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-konnectivity-ca\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:19:35.229288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.229245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.230309 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230291    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ceacc798-3509-453a-b3ad-48abc179967b-config\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:35.230309 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230301    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b0926a1-74b6-4510-8fbc-91609d959cbf-cni-binary-copy\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.230442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230319    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-script-lib\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.230442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230392    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-iptables-alerter-script\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:19:35.230581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230478    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpkt\" (UniqueName: \"kubernetes.io/projected/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-api-access-snpkt\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.230581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230627    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcbv\" (UniqueName: \"kubernetes.io/projected/ceacc798-3509-453a-b3ad-48abc179967b-kube-api-access-7wcbv\") pod \"service-ca-operator-544d749c8c-mxfmn\" (UID: \"ceacc798-3509-453a-b3ad-48abc179967b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn"
Oct 30 20:19:35.230761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230747    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d32d84-47e1-4b90-83e1-3509cd867f17-config-volume\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.230819 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.230807    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-service-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.231998    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d15d48f-5197-4096-9011-d82bcb96fde9-ovnkube-config\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232011    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d15d48f-5197-4096-9011-d82bcb96fde9-ovn-node-metrics-cert\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232018    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4840983-85d4-48ae-8de5-20aa5de9f42c-cert\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232067    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232290    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9z9\" (UniqueName: \"kubernetes.io/projected/c6d32d84-47e1-4b90-83e1-3509cd867f17-kube-api-access-7s9z9\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232356    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2md7\" (UniqueName: \"kubernetes.io/projected/c9906391-b55c-4c9c-bb13-58b6339d1f76-kube-api-access-h2md7\") pod \"migrator-66df5d496b-d6p67\" (UID: \"c9906391-b55c-4c9c-bb13-58b6339d1f76\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d6p67"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232518    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkds\" (UniqueName: \"kubernetes.io/projected/f8932125-9eb4-4f7f-b577-c232a34a0520-kube-api-access-fxkds\") pod \"network-check-source-6f78d59f99-fbzx6\" (UID: \"f8932125-9eb4-4f7f-b577-c232a34a0520\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fbzx6"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232587    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvph\" (UniqueName: \"kubernetes.io/projected/61c90821-34fa-4790-a80c-34e6fb6e02ce-kube-api-access-wwvph\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.232630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232584    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4t9\" (UniqueName: \"kubernetes.io/projected/a4840983-85d4-48ae-8de5-20aa5de9f42c-kube-api-access-ww4t9\") pod \"ingress-canary-nnc2v\" (UID: \"a4840983-85d4-48ae-8de5-20aa5de9f42c\") " pod="openshift-ingress-canary/ingress-canary-nnc2v"
Oct 30 20:19:35.232937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232779    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96f587e0-8984-434f-b1a5-bae1b56c343b-original-pull-secret\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.232937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232781    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtl7\" (UniqueName: \"kubernetes.io/projected/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-kube-api-access-mvtl7\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:19:35.232937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.232930    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f-agent-certs\") pod \"konnectivity-agent-l4ldp\" (UID: \"c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f\") " pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:19:35.233125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233106    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-config\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.233348 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233333    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfdd\" (UniqueName: \"kubernetes.io/projected/2d15d48f-5197-4096-9011-d82bcb96fde9-kube-api-access-8xfdd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.233417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233375    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh4kb\" (UniqueName: \"kubernetes.io/projected/7b7f4d68-a578-434f-894a-6cf2fbc63cda-kube-api-access-bh4kb\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.233468 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233418    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61c90821-34fa-4790-a80c-34e6fb6e02ce-serving-cert\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.233468 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233432    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfmw\" (UniqueName: \"kubernetes.io/projected/f6244e4c-72f8-40ce-9a76-2afc5467cebe-kube-api-access-hvfmw\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:19:35.233625 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233610    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.233679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233639    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-sys-fs\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.233679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233665    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-system-cni-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233688    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-conf\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233711    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-var-lib-kubelet\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233718    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233730    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-sys-fs\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233718    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-system-cni-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233734    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-kubernetes\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233768    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/2b0926a1-74b6-4510-8fbc-91609d959cbf-kube-api-access-srm2k\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.233783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233778    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-kubelet-config\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233787    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-kubernetes\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233771    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-var-lib-kubelet\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233812    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233817    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-conf\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233829    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-kubelet-config\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233836    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233866    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-run\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233888    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-cnibin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.233896    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.233910    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233916    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-device-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.233951    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.733932593 +0000 UTC m=+1.882983578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233970    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cnibin\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233970    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-device-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233981    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-run\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.233989    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-cnibin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234001    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.733979865 +0000 UTC m=+1.883030838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234000    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-cnibin\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234040    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234060    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-modprobe-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234081    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-sys\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234094    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234096    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234121    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-netd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234143    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234154    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234157    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-netd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234170    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-sys\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234169    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.234168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234199    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-hostroot\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234214    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-registration-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234218    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234228    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-modprobe-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234226    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.734213777 +0000 UTC m=+1.883264746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-registration-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234259    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-hostroot\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234275    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-systemd\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234295    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6244e4c-72f8-40ce-9a76-2afc5467cebe-hosts-file\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234306    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwlr\" (UniqueName: \"kubernetes.io/projected/7de73755-c298-441d-a185-bdaf04d2f7ea-kube-api-access-5rwlr\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234313    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234351    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6244e4c-72f8-40ce-9a76-2afc5467cebe-hosts-file\") pod \"node-resolver-b5kfh\" (UID: \"f6244e4c-72f8-40ce-9a76-2afc5467cebe\") " pod="openshift-dns/node-resolver-b5kfh"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234354    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hxx\" (UniqueName: \"kubernetes.io/projected/3084f525-41f6-44e7-9cf4-e0a85e23362e-kube-api-access-k2hxx\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234360    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234374    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234384    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234334    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-systemd\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234398    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-conf-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234414    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234429    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-system-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234438    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-conf-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234446    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-k8s-cni-cncf-io\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234444    5002 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234449    5002 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234460    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-slash\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234479    5002 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234484    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-k8s-cni-cncf-io\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234489    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-slash\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234499    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-log-socket\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234506    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.734497827 +0000 UTC m=+1.883548793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234520    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-log-socket\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234524    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-system-cni-dir\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234564    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.73455389 +0000 UTC m=+1.883604891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : secret "metrics-daemon-secret" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234582    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysconfig\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234604    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.734591205 +0000 UTC m=+1.883642175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234633    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-multus-certs\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234654    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysconfig\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234689    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-socket-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234725    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-multus-certs\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234728    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/877333f1-3281-4c9a-84be-ac1de10369e4-host\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234756    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/877333f1-3281-4c9a-84be-ac1de10369e4-host\") pod \"node-ca-x2lgj\" (UID: \"877333f1-3281-4c9a-84be-ac1de10369e4\") " pod="openshift-image-registry/node-ca-x2lgj"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234758    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234760    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-socket-dir\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234778    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-kubelet\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234794    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-bin\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234822    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-socket-dir-parent\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234828    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234840    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234845    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-systemd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234855    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-cni-bin\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234868    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-etc-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234825    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-kubelet\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.234876    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.734864595 +0000 UTC m=+1.883915575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234899    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-etc-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234899    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-multus-socket-dir-parent\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234915    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234899    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-systemd\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234940    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-var-lib-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234956    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234970    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-node-log\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234976    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpl42\" (UniqueName: \"kubernetes.io/projected/b93cda06-ee44-4b46-8a1b-1cee671fa4e7-kube-api-access-kpl42\") pod \"aws-ebs-csi-driver-node-4jm58\" (UID: \"b93cda06-ee44-4b46-8a1b-1cee671fa4e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jm58"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234984    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.234996    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-var-lib-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235007    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-netns\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235025    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-node-log\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235034    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.235045    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.735035818 +0000 UTC m=+1.884086817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.235058    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235065    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-openvswitch\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235069    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-netns\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235073    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.235092    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.735080565 +0000 UTC m=+1.884131532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235116    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-netns\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.235121    5002 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235121    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-etc-sysctl-d\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235132    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-bin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235153    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-kubelet\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.235162    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.735153515 +0000 UTC m=+1.884204494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235165    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-run-netns\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235176    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-bin\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235180    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-kubelet\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235179    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.235203    5002 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.235228    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:35.735220275 +0000 UTC m=+1.884271241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235228    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-systemd-units\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235249    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235264    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-systemd-units\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235286    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235267    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-dbus\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235303    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96f587e0-8984-434f-b1a5-bae1b56c343b-dbus\") pod \"global-pull-secret-syncer-jkspb\" (UID: \"96f587e0-8984-434f-b1a5-bae1b56c343b\") " pod="kube-system/global-pull-secret-syncer-jkspb"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235313    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-ovn\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235330    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-multus\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235345    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-crio-socket\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235361    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-etc-kubernetes\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235397    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-etc-kubernetes\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235404    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-crio-socket\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235408    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235422    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d15d48f-5197-4096-9011-d82bcb96fde9-run-ovn\") pod \"ovnkube-node-6vhjk\" (UID: \"2d15d48f-5197-4096-9011-d82bcb96fde9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235428    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b0926a1-74b6-4510-8fbc-91609d959cbf-host-var-lib-cni-multus\") pod \"multus-g6df8\" (UID: \"2b0926a1-74b6-4510-8fbc-91609d959cbf\") " pod="openshift-multus/multus-g6df8"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235428    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-host\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235453    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-lib-modules\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235469    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-host-slash\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-host\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.235406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235454    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62ad0a1a-eede-46f0-8e64-88b9d86a5239-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.238525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235514    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d003e73-1f2d-4e65-a6dc-f8ead9bc6109-host-slash\") pod \"iptables-alerter-nv9fr\" (UID: \"2d003e73-1f2d-4e65-a6dc-f8ead9bc6109\") " pod="openshift-network-operator/iptables-alerter-nv9fr"
Oct 30 20:19:35.238525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235560    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7de73755-c298-441d-a185-bdaf04d2f7ea-lib-modules\") pod \"tuned-zz2fw\" (UID: \"7de73755-c298-441d-a185-bdaf04d2f7ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zz2fw"
Oct 30 20:19:35.238525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235625    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnbn\" (UniqueName: \"kubernetes.io/projected/62ad0a1a-eede-46f0-8e64-88b9d86a5239-kube-api-access-ksnbn\") pod \"multus-additional-cni-plugins-r25dm\" (UID: \"62ad0a1a-eede-46f0-8e64-88b9d86a5239\") " pod="openshift-multus/multus-additional-cni-plugins-r25dm"
Oct 30 20:19:35.238525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235814    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 20:19:35.238525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.235877    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:19:35.240086 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.240070    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4d68-a578-434f-894a-6cf2fbc63cda-trusted-ca\") pod \"console-operator-7cd56b8dbc-jnzz2\" (UID: \"7b7f4d68-a578-434f-894a-6cf2fbc63cda\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:35.242270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.242254    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61c90821-34fa-4790-a80c-34e6fb6e02ce-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mjg55\" (UID: \"61c90821-34fa-4790-a80c-34e6fb6e02ce\") " pod="openshift-insights/insights-operator-7c4986d449-mjg55"
Oct 30 20:19:35.372227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.372204    5002 scope.go:122] "RemoveContainer" containerID="6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b"
Oct 30 20:19:35.739559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739469    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.739559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739511    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:35.739559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739560    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739596    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739615    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739631    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739640    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739649    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739658    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739707    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739689445 +0000 UTC m=+2.888740437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739709    5002 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739729    5002 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739744    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739731465 +0000 UTC m=+2.888782431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739752    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739764    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739754753 +0000 UTC m=+2.888805721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739770    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739784    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739775706 +0000 UTC m=+2.888826673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : secret "metrics-daemon-secret" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739788    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739750    5002 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739798    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:35.739800 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739806    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739794954 +0000 UTC m=+2.888845920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739825    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739818143 +0000 UTC m=+2.888869138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739845    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739869    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.739901    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739932    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739923225 +0000 UTC m=+2.888974191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739954    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739974    5002 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.739983    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739976655 +0000 UTC m=+2.889027621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.740009    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.739999467 +0000 UTC m=+2.889050434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.740008    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:35.740373 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:35.740026    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:36.740020003 +0000 UTC m=+2.889070968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:35.740742 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:35.740378    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:36.023176 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.023114    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:19:36.023525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.023462    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/1.log"
Oct 30 20:19:36.023525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.023488    5002 generic.go:356] "Generic (PLEG): container finished" podID="7b7f4d68-a578-434f-894a-6cf2fbc63cda" containerID="0c45c7a0a335c89efe7a41b3310ad2fc986a16f7725f1592911b1e36530e580f" exitCode=255
Oct 30 20:19:36.023624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.023570    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" event={"ID":"7b7f4d68-a578-434f-894a-6cf2fbc63cda","Type":"ContainerDied","Data":"0c45c7a0a335c89efe7a41b3310ad2fc986a16f7725f1592911b1e36530e580f"}
Oct 30 20:19:36.023669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.023653    5002 scope.go:122] "RemoveContainer" containerID="6fe42e257d68051469d73118c2106a1a52255b5540834141321e0675fd51e38b"
Oct 30 20:19:36.024000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.023981    5002 scope.go:122] "RemoveContainer" containerID="0c45c7a0a335c89efe7a41b3310ad2fc986a16f7725f1592911b1e36530e580f"
Oct 30 20:19:36.024135 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.024119    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jnzz2_openshift-console-operator(7b7f4d68-a578-434f-894a-6cf2fbc63cda)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" podUID="7b7f4d68-a578-434f-894a-6cf2fbc63cda"
Oct 30 20:19:36.108328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.108300    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:36.219525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.219506    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:36.329804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.329749    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:36.748168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748138    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:36.748168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748170    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:36.748334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748193    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:36.748334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748229    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:36.748334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748263    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:36.748334 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748281    5002 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:36.748334 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748281    5002 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:19:36.748334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748291    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:36.748334 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748340    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748321505 +0000 UTC m=+4.897372474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748371    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs podName:3084f525-41f6-44e7-9cf4-e0a85e23362e nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748360437 +0000 UTC m=+4.897411404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs") pod "network-metrics-daemon-svcz9" (UID: "3084f525-41f6-44e7-9cf4-e0a85e23362e") : secret "metrics-daemon-secret" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748372    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748387    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748381253 +0000 UTC m=+4.897432218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748383    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748392    5002 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748411    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748430    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748416504 +0000 UTC m=+4.897467471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748394    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4544dfb8-94842: secret "image-registry-tls" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748456    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748449174 +0000 UTC m=+4.897500139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748457    5002 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748469    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls podName:6d797780-22b8-4dee-bb0c-e2b0a38cceac nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748464 +0000 UTC m=+4.897514966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls") pod "image-registry-7f4544dfb8-94842" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac") : secret "image-registry-tls" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748489    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748480317 +0000 UTC m=+4.897531338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748563    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748584    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:36.748623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.748611    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:36.749085 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748659    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:36.749085 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748682    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:36.749085 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748704    5002 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:19:36.749085 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748715    5002 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-695dcdfbc7-z5fr7: secret "image-registry-tls" not found
Oct 30 20:19:36.749085 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748685    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748678335 +0000 UTC m=+4.897729300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:36.749085 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748738    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748725715 +0000 UTC m=+4.897776687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:36.749085 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:36.748754    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls podName:66fe23b4-c77d-49af-95de-6e87cc4eb223 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.748744449 +0000 UTC m=+4.897795418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls") pod "image-registry-695dcdfbc7-z5fr7" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223") : secret "image-registry-tls" not found
Oct 30 20:19:36.869858 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:36.869833    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:37.029017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:37.028960    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:19:37.183635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:37.183612    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:37.405674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:37.405610    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:37.589889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:37.589864    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:37.841472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:37.841450    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:37.972765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:37.972740    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:37.973025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:37.973012    5002 scope.go:122] "RemoveContainer" containerID="0c45c7a0a335c89efe7a41b3310ad2fc986a16f7725f1592911b1e36530e580f"
Oct 30 20:19:37.973166 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:37.973149    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jnzz2_openshift-console-operator(7b7f4d68-a578-434f-894a-6cf2fbc63cda)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" podUID="7b7f4d68-a578-434f-894a-6cf2fbc63cda"
Oct 30 20:19:38.325217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.325185    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:38.325614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.325384    5002 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:38.463624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.463597    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:38.684679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.684606    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:38.766827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.766799    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:38.766949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.766834    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:38.766949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.766868    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:38.766949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.766903    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:38.766949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.766927    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.766951    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.766954    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.766990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.766951    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767011    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.766992856 +0000 UTC m=+8.916043842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767029    5002 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.767049    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767073    5002 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767080    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.767063874 +0000 UTC m=+8.916114857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : secret "router-metrics-certs-default" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767109    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.767099483 +0000 UTC m=+8.916150454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767108    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:38.767143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.767142    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:38.767657 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767153    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert podName:71f15b82-a100-493b-aef0-c83d07ad1806 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.767136409 +0000 UTC m=+8.916187378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hdjsr" (UID: "71f15b82-a100-493b-aef0-c83d07ad1806") : secret "networking-console-plugin-cert" not found
Oct 30 20:19:38.767657 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767185    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.767170089 +0000 UTC m=+8.916221062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:38.767657 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767211    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.767202005 +0000 UTC m=+8.916252972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:38.767657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.767241    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:38.767657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.767266    5002 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:19:38.767657 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767327    5002 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:19:38.767657 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:38.767362    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls podName:2f260119-6cc6-44c8-b7e7-c8c6ae3471fc nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.767355821 +0000 UTC m=+8.916406807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t676" (UID: "2f260119-6cc6-44c8-b7e7-c8c6ae3471fc") : secret "samples-operator-tls" not found
Oct 30 20:19:38.769519 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.769488    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3084f525-41f6-44e7-9cf4-e0a85e23362e-metrics-certs\") pod \"network-metrics-daemon-svcz9\" (UID: \"3084f525-41f6-44e7-9cf4-e0a85e23362e\") " pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:38.769635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.769579    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"image-registry-7f4544dfb8-94842\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") " pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:38.769687 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.769636    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"image-registry-695dcdfbc7-z5fr7\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") " pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:38.916787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.916762    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:38.969839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.969817    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:38.976093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.976070    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svcz9"
Oct 30 20:19:38.982258 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:38.982245    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:39.049125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.049109    5002 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:39.049428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.049415    5002 scope.go:122] "RemoveContainer" containerID="0c45c7a0a335c89efe7a41b3310ad2fc986a16f7725f1592911b1e36530e580f"
Oct 30 20:19:39.049587 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:39.049569    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jnzz2_openshift-console-operator(7b7f4d68-a578-434f-894a-6cf2fbc63cda)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" podUID="7b7f4d68-a578-434f-894a-6cf2fbc63cda"
Oct 30 20:19:39.118317 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.118293    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:39.309673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.309614    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:39.489024 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.488993    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:39.513674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.513651    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svcz9"]
Oct 30 20:19:39.517213 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:39.517183    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3084f525_41f6_44e7_9cf4_e0a85e23362e.slice/crio-1b2354c23b3e5e7c80a494a20ae7595c6f134db76961fe2d21ae16f22cfb6e19 WatchSource:0}: Error finding container 1b2354c23b3e5e7c80a494a20ae7595c6f134db76961fe2d21ae16f22cfb6e19: Status 404 returned error can't find the container with id 1b2354c23b3e5e7c80a494a20ae7595c6f134db76961fe2d21ae16f22cfb6e19
Oct 30 20:19:39.519375 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.519356    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:19:39.532078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.532046    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4544dfb8-94842"]
Oct 30 20:19:39.534685 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:39.534658    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d797780_22b8_4dee_bb0c_e2b0a38cceac.slice/crio-7f7fa1ec06a8c3e087afa32cd3a5a14f7c2af8453a3bcc11f6fb3ca7e7e401f0 WatchSource:0}: Error finding container 7f7fa1ec06a8c3e087afa32cd3a5a14f7c2af8453a3bcc11f6fb3ca7e7e401f0: Status 404 returned error can't find the container with id 7f7fa1ec06a8c3e087afa32cd3a5a14f7c2af8453a3bcc11f6fb3ca7e7e401f0
Oct 30 20:19:39.535444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.535418    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-695dcdfbc7-z5fr7"]
Oct 30 20:19:39.562661 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:39.562641    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fe23b4_c77d_49af_95de_6e87cc4eb223.slice/crio-118dbebbaeed91380ce40f5df73d4d0fb78b6608396e3574fbf8893a91cafd1c WatchSource:0}: Error finding container 118dbebbaeed91380ce40f5df73d4d0fb78b6608396e3574fbf8893a91cafd1c: Status 404 returned error can't find the container with id 118dbebbaeed91380ce40f5df73d4d0fb78b6608396e3574fbf8893a91cafd1c
Oct 30 20:19:39.659969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.659922    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:39.844333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:39.844265    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:40.039407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.039372    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" event={"ID":"6d797780-22b8-4dee-bb0c-e2b0a38cceac","Type":"ContainerStarted","Data":"a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb"}
Oct 30 20:19:40.039585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.039422    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" event={"ID":"6d797780-22b8-4dee-bb0c-e2b0a38cceac","Type":"ContainerStarted","Data":"7f7fa1ec06a8c3e087afa32cd3a5a14f7c2af8453a3bcc11f6fb3ca7e7e401f0"}
Oct 30 20:19:40.040485 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.040461    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svcz9" event={"ID":"3084f525-41f6-44e7-9cf4-e0a85e23362e","Type":"ContainerStarted","Data":"1b2354c23b3e5e7c80a494a20ae7595c6f134db76961fe2d21ae16f22cfb6e19"}
Oct 30 20:19:40.041827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.041805    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7" event={"ID":"66fe23b4-c77d-49af-95de-6e87cc4eb223","Type":"ContainerStarted","Data":"715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88"}
Oct 30 20:19:40.041917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.041846    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7" event={"ID":"66fe23b4-c77d-49af-95de-6e87cc4eb223","Type":"ContainerStarted","Data":"118dbebbaeed91380ce40f5df73d4d0fb78b6608396e3574fbf8893a91cafd1c"}
Oct 30 20:19:40.155910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.155857    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b5kfh_f6244e4c-72f8-40ce-9a76-2afc5467cebe/dns-node-resolver/0.log"
Oct 30 20:19:40.720929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.720899    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:40.820177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.820144    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-695dcdfbc7-z5fr7_66fe23b4-c77d-49af-95de-6e87cc4eb223/registry/0.log"
Oct 30 20:19:40.956743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:40.956718    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f4544dfb8-94842_6d797780-22b8-4dee-bb0c-e2b0a38cceac/registry/0.log"
Oct 30 20:19:41.044944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:41.044880    5002 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:41.045088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:41.044881    5002 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:41.457486 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:41.457460    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:19:41.556590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:41.556522    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x2lgj_877333f1-3281-4c9a-84be-ac1de10369e4/node-ca/0.log"
Oct 30 20:19:41.658339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:41.658316    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l4ldp"
Oct 30 20:19:42.053307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.053266    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svcz9" event={"ID":"3084f525-41f6-44e7-9cf4-e0a85e23362e","Type":"ContainerStarted","Data":"526bb80557233b7d1a400190e912f667efa839da386cc16f7b562022e3146a6a"}
Oct 30 20:19:42.053693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.053362    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svcz9" event={"ID":"3084f525-41f6-44e7-9cf4-e0a85e23362e","Type":"ContainerStarted","Data":"b9245fa914df1d59c7adbd000199ce88e0258577a7729d000af0c585592c9853"}
Oct 30 20:19:42.157025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.157000    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nnc2v_a4840983-85d4-48ae-8de5-20aa5de9f42c/serve-healthcheck-canary/0.log"
Oct 30 20:19:42.356697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.356638    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-d6p67_c9906391-b55c-4c9c-bb13-58b6339d1f76/migrator/0.log"
Oct 30 20:19:42.557540 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.557512    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-qlgnw_d34f102b-c1f4-4e19-a7ce-cdc1ea51930f/migrator/0.log"
Oct 30 20:19:42.740077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.740043    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:19:42.758115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.758089    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7_70698df1-1b21-49a3-9cb1-131b2b113024/kube-storage-version-migrator-operator/0.log"
Oct 30 20:19:42.804590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.804558    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:42.804750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.804605    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:42.804750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.804666    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:42.804750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.804699    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:42.804750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.804733    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:42.805041 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.804868    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:42.805041 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.804898    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:42.805180 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:42.804869    5002 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:42.805328 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:42.805028    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle podName:2dfc646e-d42e-44ad-8a5c-09e8115c06ca nodeName:}" failed. No retries permitted until 2026-10-30 20:19:50.805008302 +0000 UTC m=+16.954059286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle") pod "router-default-67cdd9f98c-zf64k" (UID: "2dfc646e-d42e-44ad-8a5c-09e8115c06ca") : configmap references non-existent config key: service-ca.crt
Oct 30 20:19:42.805328 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:42.805218    5002 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:42.805328 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:42.805239    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls podName:c6d32d84-47e1-4b90-83e1-3509cd867f17 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:50.805222988 +0000 UTC m=+16.954273959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls") pod "dns-default-8hmbp" (UID: "c6d32d84-47e1-4b90-83e1-3509cd867f17") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:42.805328 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:42.805138    5002 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:42.805328 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:42.805269    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls podName:0bb8ae2c-9126-4e64-b930-c748b4a44bf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:50.805256285 +0000 UTC m=+16.954307287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9v6j5" (UID: "0bb8ae2c-9126-4e64-b930-c748b4a44bf1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:19:42.805328 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:19:42.805285    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls podName:5687ec3f-b05f-4e0b-b4ce-04f50516b5d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:50.805277035 +0000 UTC m=+16.954328000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cmnm9" (UID: "5687ec3f-b05f-4e0b-b4ce-04f50516b5d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:42.807369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.807339    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-metrics-certs\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:42.807750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.807728    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f15b82-a100-493b-aef0-c83d07ad1806-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hdjsr\" (UID: \"71f15b82-a100-493b-aef0-c83d07ad1806\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:42.807834 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.807765    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f260119-6cc6-44c8-b7e7-c8c6ae3471fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t676\" (UID: \"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:42.868694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.868664    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"
Oct 30 20:19:42.872918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:42.872900    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"
Oct 30 20:19:43.308779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:43.308749    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hdjsr"]
Oct 30 20:19:43.311777 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:43.311752    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f15b82_a100_493b_aef0_c83d07ad1806.slice/crio-ef15ae190b26d3366797f05357d1ffe9bdefc79526899c7862e09e1256e1bbb0 WatchSource:0}: Error finding container ef15ae190b26d3366797f05357d1ffe9bdefc79526899c7862e09e1256e1bbb0: Status 404 returned error can't find the container with id ef15ae190b26d3366797f05357d1ffe9bdefc79526899c7862e09e1256e1bbb0
Oct 30 20:19:43.315566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:43.315519    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676"]
Oct 30 20:19:44.064293 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:44.064255    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676" event={"ID":"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc","Type":"ContainerStarted","Data":"d5bbf24a723d8368755bc1e583338040126ae44074eac11570c8fd772d0f6db6"}
Oct 30 20:19:44.065584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:44.065553    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr" event={"ID":"71f15b82-a100-493b-aef0-c83d07ad1806","Type":"ContainerStarted","Data":"ef15ae190b26d3366797f05357d1ffe9bdefc79526899c7862e09e1256e1bbb0"}
Oct 30 20:19:44.313690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:44.313662    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vhjk"
Oct 30 20:19:44.694557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:44.693100    5002 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-29.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:19:44.924477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:44.924441    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:45.153997 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:45.153963    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:45.327617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:45.327590    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:45.519803 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:45.519737    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:45.941693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:45.941662    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:19:45.959475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:45.959448    5002 scope.go:122] "RemoveContainer" containerID="0c45c7a0a335c89efe7a41b3310ad2fc986a16f7725f1592911b1e36530e580f"
Oct 30 20:19:46.074798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.074720    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-hdjsr" event={"ID":"71f15b82-a100-493b-aef0-c83d07ad1806","Type":"ContainerStarted","Data":"25c600aa8e4b1b85e71a4b7abcf01ea72772b7a3890d90baf94656a9f7d7dbd4"}
Oct 30 20:19:46.076577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.076556    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:19:46.076694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.076627    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" event={"ID":"7b7f4d68-a578-434f-894a-6cf2fbc63cda","Type":"ContainerStarted","Data":"886e0ddf37ea6bfe5b97849dbe825c091dcf164cf9627352bef9a3a47c213057"}
Oct 30 20:19:46.076912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.076894    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:46.077824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.077800    5002 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jnzz2 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Oct 30 20:19:46.077910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.077849    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2" podUID="7b7f4d68-a578-434f-894a-6cf2fbc63cda" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused"
Oct 30 20:19:46.715303 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.715259    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:46.924291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:46.924216    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:47.082050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.081965    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676" event={"ID":"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc","Type":"ContainerStarted","Data":"dad89940440e4c7a8a84f7795caa4fe6abfe078f98259dfbc7f576b5eaf86c65"}
Oct 30 20:19:47.082282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.082062    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t676" event={"ID":"2f260119-6cc6-44c8-b7e7-c8c6ae3471fc","Type":"ContainerStarted","Data":"1e12ade4dc8f653a47b083a7cadab04defa961daa0ca889edab5c60b03c17280"}
Oct 30 20:19:47.085971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.085949    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jnzz2"
Oct 30 20:19:47.106869 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.106832    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:47.316753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.316712    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:47.545654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.545623    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:47.727401 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.727352    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:47.911862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:47.911803    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:48.116750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:48.116720    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:50.873335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.873306    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:50.873335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.873338    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:50.873759 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.873380    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:50.873759 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.873407    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:50.874731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.874581    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dfc646e-d42e-44ad-8a5c-09e8115c06ca-service-ca-bundle\") pod \"router-default-67cdd9f98c-zf64k\" (UID: \"2dfc646e-d42e-44ad-8a5c-09e8115c06ca\") " pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:50.876817 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.876790    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6d32d84-47e1-4b90-83e1-3509cd867f17-metrics-tls\") pod \"dns-default-8hmbp\" (UID: \"c6d32d84-47e1-4b90-83e1-3509cd867f17\") " pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:50.877566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.876969    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb8ae2c-9126-4e64-b930-c748b4a44bf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9v6j5\" (UID: \"0bb8ae2c-9126-4e64-b930-c748b4a44bf1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:50.877566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.876884    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5687ec3f-b05f-4e0b-b4ce-04f50516b5d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cmnm9\" (UID: \"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0\") " pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:50.969173 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.969150    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:50.970216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.970198    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"
Oct 30 20:19:50.982576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.982552    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:50.984011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:50.983995    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cmnm9"
Oct 30 20:19:51.477309 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:51.477278    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5"]
Oct 30 20:19:51.480669 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:51.480592    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb8ae2c_9126_4e64_b930_c748b4a44bf1.slice/crio-e807d858b4ce6eb6e52f9e43f27d4f598f6d238a4b645887762ed1288db90279 WatchSource:0}: Error finding container e807d858b4ce6eb6e52f9e43f27d4f598f6d238a4b645887762ed1288db90279: Status 404 returned error can't find the container with id e807d858b4ce6eb6e52f9e43f27d4f598f6d238a4b645887762ed1288db90279
Oct 30 20:19:51.482747 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:51.482724    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfc646e_d42e_44ad_8a5c_09e8115c06ca.slice/crio-bb836d4e6ac471f21a4c235ecaf91be20c2b6195ea08bf5464438001767be136 WatchSource:0}: Error finding container bb836d4e6ac471f21a4c235ecaf91be20c2b6195ea08bf5464438001767be136: Status 404 returned error can't find the container with id bb836d4e6ac471f21a4c235ecaf91be20c2b6195ea08bf5464438001767be136
Oct 30 20:19:51.528308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:51.528287    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67cdd9f98c-zf64k"]
Oct 30 20:19:51.529229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:51.529208    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cmnm9"]
Oct 30 20:19:51.529660 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:51.529639    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d32d84_47e1_4b90_83e1_3509cd867f17.slice/crio-5d622acf9889071dd616c3e486869ed15bd5e09b17a2d93ec86e29dd94ef4d0e WatchSource:0}: Error finding container 5d622acf9889071dd616c3e486869ed15bd5e09b17a2d93ec86e29dd94ef4d0e: Status 404 returned error can't find the container with id 5d622acf9889071dd616c3e486869ed15bd5e09b17a2d93ec86e29dd94ef4d0e
Oct 30 20:19:51.530365 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:19:51.530340    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5687ec3f_b05f_4e0b_b4ce_04f50516b5d0.slice/crio-4dc9281463050ad81235288f0369cb349aa70dcff79ab8d2eb7b3e51b07344d3 WatchSource:0}: Error finding container 4dc9281463050ad81235288f0369cb349aa70dcff79ab8d2eb7b3e51b07344d3: Status 404 returned error can't find the container with id 4dc9281463050ad81235288f0369cb349aa70dcff79ab8d2eb7b3e51b07344d3
Oct 30 20:19:51.530782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:51.530767    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hmbp"]
Oct 30 20:19:52.105837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.105797    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5" event={"ID":"0bb8ae2c-9126-4e64-b930-c748b4a44bf1","Type":"ContainerStarted","Data":"e807d858b4ce6eb6e52f9e43f27d4f598f6d238a4b645887762ed1288db90279"}
Oct 30 20:19:52.107491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.107464    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hmbp" event={"ID":"c6d32d84-47e1-4b90-83e1-3509cd867f17","Type":"ContainerStarted","Data":"5d622acf9889071dd616c3e486869ed15bd5e09b17a2d93ec86e29dd94ef4d0e"}
Oct 30 20:19:52.109131 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.109099    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cmnm9" event={"ID":"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0","Type":"ContainerStarted","Data":"be9cd7b1a05df378d5afe80e81f0c9e8d36c6d6669b3da55a1f219698aaf8067"}
Oct 30 20:19:52.109241 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.109151    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cmnm9" event={"ID":"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0","Type":"ContainerStarted","Data":"4dc9281463050ad81235288f0369cb349aa70dcff79ab8d2eb7b3e51b07344d3"}
Oct 30 20:19:52.110644 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.110623    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67cdd9f98c-zf64k" event={"ID":"2dfc646e-d42e-44ad-8a5c-09e8115c06ca","Type":"ContainerStarted","Data":"ed4ee9bfdadb556449dacacba5d35c188891ba4dda03905a45a452528021af91"}
Oct 30 20:19:52.110740 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.110661    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67cdd9f98c-zf64k" event={"ID":"2dfc646e-d42e-44ad-8a5c-09e8115c06ca","Type":"ContainerStarted","Data":"bb836d4e6ac471f21a4c235ecaf91be20c2b6195ea08bf5464438001767be136"}
Oct 30 20:19:52.506913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.506882    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-695dcdfbc7-z5fr7"]
Oct 30 20:19:52.810594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.810493    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:52.970028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.969992    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:52.973108 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:52.973083    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:53.116340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:53.116268    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:53.117583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:53.117561    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67cdd9f98c-zf64k"
Oct 30 20:19:53.193809 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:53.193779    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:53.410525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:53.410497    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:54.121049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:54.121017    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cmnm9" event={"ID":"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0","Type":"ContainerStarted","Data":"30370c58a6e8842919af0bf074b375f6e1464bfd2f727b4fda7195c25487e836"}
Oct 30 20:19:55.127499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:55.127410    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9v6j5" event={"ID":"0bb8ae2c-9126-4e64-b930-c748b4a44bf1","Type":"ContainerStarted","Data":"e203976e49dd3b8f220a9604c39f78d838a2e194c56de9a3e79e1154ef1b8047"}
Oct 30 20:19:55.130101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:55.130068    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hmbp" event={"ID":"c6d32d84-47e1-4b90-83e1-3509cd867f17","Type":"ContainerStarted","Data":"f8f6eb78fc93a31005421ab3d02a5006823d59b4bfac0453f0c746efc7aa69da"}
Oct 30 20:19:55.130239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:55.130126    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hmbp" event={"ID":"c6d32d84-47e1-4b90-83e1-3509cd867f17","Type":"ContainerStarted","Data":"12dc6509e6e1e37b00c5b14f9946a89887cb4dfc641297d5996b5f4f828cac7e"}
Oct 30 20:19:55.849782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:55.849707    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:55.946428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:55.946397    5002 patch_prober.go:29] interesting pod/image-registry-7f4544dfb8-94842 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:19:55.946569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:55.946438    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:19:56.123446 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:56.123389    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:56.135377 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:56.135346    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cmnm9" event={"ID":"5687ec3f-b05f-4e0b-b4ce-04f50516b5d0","Type":"ContainerStarted","Data":"189f6e242bf1f63d8b779156fb98dddc74fb6d7044053ee635652f143d7954cc"}
Oct 30 20:19:56.135807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:56.135792    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:19:56.379066 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:56.379000    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:56.897172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:56.897140    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:57.141048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:57.141015    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:58.893710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:58.893679    5002 patch_prober.go:29] interesting pod/image-registry-7f4544dfb8-94842 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:19:58.894181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:19:58.893733    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:20:00.168296 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:00.168259    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:00.338876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:00.338834    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:00.518156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:00.518127    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:00.759711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:00.759685    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:01.459965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:01.459939    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n6f9w"
Oct 30 20:20:01.995088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:01.995051    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:02.179643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:02.179616    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:02.512614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:02.512588    5002 patch_prober.go:29] interesting pod/image-registry-695dcdfbc7-z5fr7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:20:02.512994 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:02.512649    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7" podUID="66fe23b4-c77d-49af-95de-6e87cc4eb223" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:20:05.813654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:05.813611    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:05.946380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:05.946346    5002 patch_prober.go:29] interesting pod/image-registry-7f4544dfb8-94842 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:20:05.946558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:05.946407    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:20:06.141258 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:06.141196    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8hmbp"
Oct 30 20:20:06.378109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:06.378050    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:06.606568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:06.606519    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:06.810654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:06.810625    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:06.998105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:06.998080    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:07.106212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.106184    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hg6rr"]
Oct 30 20:20:07.109274 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.109250    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.114686 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:20:07.114656    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:20:07.174597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.174568    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:07.190412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190376    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-sys\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190410    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72869d66-7380-4336-92f3-fd98774219de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190486    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72869d66-7380-4336-92f3-fd98774219de-metrics-client-ca\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190515    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72869d66-7380-4336-92f3-fd98774219de-node-exporter-textfile\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190571    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89c5\" (UniqueName: \"kubernetes.io/projected/72869d66-7380-4336-92f3-fd98774219de-kube-api-access-w89c5\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190680    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-node-exporter-wtmp\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190716    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72869d66-7380-4336-92f3-fd98774219de-node-exporter-tls\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190753    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-root\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.190914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.190773    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72869d66-7380-4336-92f3-fd98774219de-node-exporter-accelerators-collector-config\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.231164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.231142    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:20:07.231289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.231222    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9jp6\""
Oct 30 20:20:07.231356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.231325    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:20:07.231356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.231334    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:20:07.231433 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.231384    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:20:07.292064 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292011    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72869d66-7380-4336-92f3-fd98774219de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292064 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292053    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72869d66-7380-4336-92f3-fd98774219de-metrics-client-ca\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292078    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72869d66-7380-4336-92f3-fd98774219de-node-exporter-textfile\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292097    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89c5\" (UniqueName: \"kubernetes.io/projected/72869d66-7380-4336-92f3-fd98774219de-kube-api-access-w89c5\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292115    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-node-exporter-wtmp\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292133    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72869d66-7380-4336-92f3-fd98774219de-node-exporter-tls\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292278    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-root\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292315    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72869d66-7380-4336-92f3-fd98774219de-node-exporter-accelerators-collector-config\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292352    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-root\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292380    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-sys\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292315    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-node-exporter-wtmp\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292466    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72869d66-7380-4336-92f3-fd98774219de-sys\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292471    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72869d66-7380-4336-92f3-fd98774219de-node-exporter-textfile\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292768    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72869d66-7380-4336-92f3-fd98774219de-metrics-client-ca\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.292926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.292906    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72869d66-7380-4336-92f3-fd98774219de-node-exporter-accelerators-collector-config\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.295172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.295150    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72869d66-7380-4336-92f3-fd98774219de-node-exporter-tls\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.295243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.295151    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72869d66-7380-4336-92f3-fd98774219de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.348440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.348416    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89c5\" (UniqueName: \"kubernetes.io/projected/72869d66-7380-4336-92f3-fd98774219de-kube-api-access-w89c5\") pod \"node-exporter-hg6rr\" (UID: \"72869d66-7380-4336-92f3-fd98774219de\") " pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.354669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.354646    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:07.421918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.421898    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hg6rr"
Oct 30 20:20:07.432514 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:20:07.432478    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72869d66_7380_4336_92f3_fd98774219de.slice/crio-71f85b87f651ddbfd81f7448a9fc1917a7de9ca426d0f67bd708db28ba38b273 WatchSource:0}: Error finding container 71f85b87f651ddbfd81f7448a9fc1917a7de9ca426d0f67bd708db28ba38b273: Status 404 returned error can't find the container with id 71f85b87f651ddbfd81f7448a9fc1917a7de9ca426d0f67bd708db28ba38b273
Oct 30 20:20:07.582491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:07.582444    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:08.181976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:08.181941    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6rr" event={"ID":"72869d66-7380-4336-92f3-fd98774219de","Type":"ContainerStarted","Data":"71f85b87f651ddbfd81f7448a9fc1917a7de9ca426d0f67bd708db28ba38b273"}
Oct 30 20:20:08.493245 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:08.493212    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:08.724053 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:08.723987    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:08.894588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:08.894564    5002 patch_prober.go:29] interesting pod/image-registry-7f4544dfb8-94842 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:20:08.894699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:08.894604    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:20:09.187699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:09.187632    5002 generic.go:356] "Generic (PLEG): container finished" podID="72869d66-7380-4336-92f3-fd98774219de" containerID="7788c17a666767853a6246af3bea2fb7ca966358ea594ed78843b64b5bfd1a5c" exitCode=0
Oct 30 20:20:09.188025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:09.187721    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6rr" event={"ID":"72869d66-7380-4336-92f3-fd98774219de","Type":"ContainerDied","Data":"7788c17a666767853a6246af3bea2fb7ca966358ea594ed78843b64b5bfd1a5c"}
Oct 30 20:20:09.373434 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:09.373405    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:10.194091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:10.194057    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6rr" event={"ID":"72869d66-7380-4336-92f3-fd98774219de","Type":"ContainerStarted","Data":"a075f984e4180da9a5d66a670639171fba5313399dac6f302e0c34f81169423f"}
Oct 30 20:20:10.194596 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:10.194107    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6rr" event={"ID":"72869d66-7380-4336-92f3-fd98774219de","Type":"ContainerStarted","Data":"fd3107663406a774b781283d149fe2d6fd6729ae8ccf97fd0d15780d9eb68d71"}
Oct 30 20:20:10.364843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:10.364794    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hg6rr" podStartSLOduration=1.921427484 podStartE2EDuration="3.364780366s" totalImagesPullingTime="1.370339761s" totalInitContainerRuntime="73.013121ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:10.364078513 +0000 UTC m=+36.513129495" watchObservedRunningTime="2026-10-30 20:20:10.364780366 +0000 UTC m=+36.513831354"
Oct 30 20:20:10.483810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:10.483779    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:10.669093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:10.669059    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:10.852364 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:10.852297    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:11.254360 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:11.254325    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:11.559697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:11.559635    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:11.774645 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:11.774616    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:11.978505 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:11.978481    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:12.512189 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:12.512159    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:20:14.791318 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:14.791284    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4544dfb8-94842"]
Oct 30 20:20:14.795469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:14.795447    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:20:14.941380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:14.941351    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:15.143993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:15.143928    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:15.530117 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:15.530076    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:16.199925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:16.199885    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:16.383247 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:16.383221    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:16.965075 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:16.965042    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:17.183098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.183067    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:17.527810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.527778    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7" podUID="66fe23b4-c77d-49af-95de-6e87cc4eb223" containerName="registry" containerID="cri-o://715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88" gracePeriod=30
Oct 30 20:20:17.771288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.771266    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:20:17.873922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.873861    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.873922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.873896    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.874092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.873933    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.874092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.873955    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.874092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.873977    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.874092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.874008    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.874092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.874039    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.874092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.874072    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\") pod \"66fe23b4-c77d-49af-95de-6e87cc4eb223\" (UID: \"66fe23b4-c77d-49af-95de-6e87cc4eb223\") "
Oct 30 20:20:17.874394 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.874325    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:17.875215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.875150    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:17.876787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.876722    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:17.877718 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.877672    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "kube-api-access-5sc4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:17.877934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.877910    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:17.878022 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.877979    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:17.878126 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.878103    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:17.881424 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.881405    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted" pod "66fe23b4-c77d-49af-95de-6e87cc4eb223" (UID: "66fe23b4-c77d-49af-95de-6e87cc4eb223"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:17.974797 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974778    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sc4x\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-kube-api-access-5sc4x\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:17.974797 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974797    5002 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-certificates\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:17.974922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974807    5002 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-bound-sa-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:17.974922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974817    5002 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-installation-pull-secrets\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:17.974922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974826    5002 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66fe23b4-c77d-49af-95de-6e87cc4eb223-registry-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:17.974922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974834    5002 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66fe23b4-c77d-49af-95de-6e87cc4eb223-image-registry-private-configuration\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:17.974922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974844    5002 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66fe23b4-c77d-49af-95de-6e87cc4eb223-trusted-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:17.974922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:17.974851    5002 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66fe23b4-c77d-49af-95de-6e87cc4eb223-ca-trust-extracted\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:18.225758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.225731    5002 generic.go:356] "Generic (PLEG): container finished" podID="66fe23b4-c77d-49af-95de-6e87cc4eb223" containerID="715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88" exitCode=0
Oct 30 20:20:18.225881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.225774    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7"
Oct 30 20:20:18.225881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.225802    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7" event={"ID":"66fe23b4-c77d-49af-95de-6e87cc4eb223","Type":"ContainerDied","Data":"715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88"}
Oct 30 20:20:18.225881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.225837    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695dcdfbc7-z5fr7" event={"ID":"66fe23b4-c77d-49af-95de-6e87cc4eb223","Type":"ContainerDied","Data":"118dbebbaeed91380ce40f5df73d4d0fb78b6608396e3574fbf8893a91cafd1c"}
Oct 30 20:20:18.225881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.225857    5002 scope.go:122] "RemoveContainer" containerID="715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88"
Oct 30 20:20:18.234738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.234722    5002 scope.go:122] "RemoveContainer" containerID="715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88"
Oct 30 20:20:18.235032 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:20:18.235007    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88\": container with ID starting with 715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88 not found: ID does not exist" containerID="715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88"
Oct 30 20:20:18.235115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.235038    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88"} err="failed to get container status \"715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88\": rpc error: code = NotFound desc = could not find container \"715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88\": container with ID starting with 715ee3d06797558de1bbb07c1503774cdfdfe9c77bd01e0fd18ba68f9ac31e88 not found: ID does not exist"
Oct 30 20:20:18.397102 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.397077    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-695dcdfbc7-z5fr7"]
Oct 30 20:20:18.447433 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:18.447410    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-695dcdfbc7-z5fr7"]
Oct 30 20:20:19.964107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:19.964075    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fe23b4-c77d-49af-95de-6e87cc4eb223" path="/var/lib/kubelet/pods/66fe23b4-c77d-49af-95de-6e87cc4eb223/volumes"
Oct 30 20:20:21.870670 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:21.870631    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:22.086327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:22.086298    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:22.356857 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:22.356824    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:23.512739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:23.512707    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:23.723451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:23.723425    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:24.316807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:24.316777    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:24.557140 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:24.557107    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:29.452648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:29.452602    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:30.074415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:30.074374    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:31.972824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:31.972779    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:32.160553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:32.160495    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:35.210833 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:35.210793    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:35.455190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:35.455155    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:37.118358 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:20:37.118318    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:20:39.811105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:39.811067    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry" containerID="cri-o://a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb" gracePeriod=30
Oct 30 20:20:40.045525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.045497    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:20:40.231807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.231779    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.231961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.231811    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.231961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.231838    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.232082 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.231959    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.232082 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.232003    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.232082 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.232041    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.232082 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.232061    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.232287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.232115    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\") pod \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\" (UID: \"6d797780-22b8-4dee-bb0c-e2b0a38cceac\") "
Oct 30 20:20:40.232448 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.232403    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:40.232595 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.232560    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:40.235285 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.235261    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "kube-api-access-t6j59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:40.235374 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.235260    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:40.235374 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.235295    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:40.235523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.235502    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:40.235623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.235602    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:40.240422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.240382    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted" pod "6d797780-22b8-4dee-bb0c-e2b0a38cceac" (UID: "6d797780-22b8-4dee-bb0c-e2b0a38cceac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:40.309978 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.309955    5002 generic.go:356] "Generic (PLEG): container finished" podID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerID="a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb" exitCode=0
Oct 30 20:20:40.310091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.310000    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4544dfb8-94842"
Oct 30 20:20:40.310091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.310052    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" event={"ID":"6d797780-22b8-4dee-bb0c-e2b0a38cceac","Type":"ContainerDied","Data":"a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb"}
Oct 30 20:20:40.310171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.310094    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4544dfb8-94842" event={"ID":"6d797780-22b8-4dee-bb0c-e2b0a38cceac","Type":"ContainerDied","Data":"7f7fa1ec06a8c3e087afa32cd3a5a14f7c2af8453a3bcc11f6fb3ca7e7e401f0"}
Oct 30 20:20:40.310171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.310116    5002 scope.go:122] "RemoveContainer" containerID="a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb"
Oct 30 20:20:40.318804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.318778    5002 scope.go:122] "RemoveContainer" containerID="a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb"
Oct 30 20:20:40.319056 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:20:40.319037    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb\": container with ID starting with a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb not found: ID does not exist" containerID="a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb"
Oct 30 20:20:40.319098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.319064    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb"} err="failed to get container status \"a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb\": rpc error: code = NotFound desc = could not find container \"a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb\": container with ID starting with a7162ed468dac27ee93170f879a03f3a723766d15af05efe1bd0afdde5ee0ffb not found: ID does not exist"
Oct 30 20:20:40.333391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333373    5002 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-trusted-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.333447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333391    5002 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d797780-22b8-4dee-bb0c-e2b0a38cceac-ca-trust-extracted\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.333447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333401    5002 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.333447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333410    5002 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d797780-22b8-4dee-bb0c-e2b0a38cceac-registry-certificates\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.333447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333419    5002 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-installation-pull-secrets\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.333447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333427    5002 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-bound-sa-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.333447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333436    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6j59\" (UniqueName: \"kubernetes.io/projected/6d797780-22b8-4dee-bb0c-e2b0a38cceac-kube-api-access-t6j59\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.333447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.333445    5002 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d797780-22b8-4dee-bb0c-e2b0a38cceac-image-registry-private-configuration\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.476343 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.476322    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4544dfb8-94842"]
Oct 30 20:20:40.479148 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:40.479130    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f4544dfb8-94842"]
Oct 30 20:20:41.963816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:41.963769    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" path="/var/lib/kubelet/pods/6d797780-22b8-4dee-bb0c-e2b0a38cceac/volumes"
Oct 30 20:20:44.327903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:44.327869    5002 generic.go:356] "Generic (PLEG): container finished" podID="61c90821-34fa-4790-a80c-34e6fb6e02ce" containerID="80dcc95db37a5d404f48f5c075ee1b63491012c968f5f8afe048df7e64b9f25b" exitCode=0
Oct 30 20:20:44.328239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:44.327927    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mjg55" event={"ID":"61c90821-34fa-4790-a80c-34e6fb6e02ce","Type":"ContainerDied","Data":"80dcc95db37a5d404f48f5c075ee1b63491012c968f5f8afe048df7e64b9f25b"}
Oct 30 20:20:44.328239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:44.328201    5002 scope.go:122] "RemoveContainer" containerID="80dcc95db37a5d404f48f5c075ee1b63491012c968f5f8afe048df7e64b9f25b"
Oct 30 20:20:45.333861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:45.333828    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mjg55" event={"ID":"61c90821-34fa-4790-a80c-34e6fb6e02ce","Type":"ContainerStarted","Data":"2a7d0f324dc075810ccc2bdcebc093fbdf72f283871ad858092acb95bf230228"}
Oct 30 20:20:45.436319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:45.436254    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:45.623825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:45.623747    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:46.973969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:46.973926    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:47.168818 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:47.168784    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:54.369290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:54.369259    5002 generic.go:356] "Generic (PLEG): container finished" podID="70698df1-1b21-49a3-9cb1-131b2b113024" containerID="4de31d38d5970e8ab7991252cd1146b339371b3b1d89ed9ae5de1f87023938c7" exitCode=0
Oct 30 20:20:54.369728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:54.369296    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7" event={"ID":"70698df1-1b21-49a3-9cb1-131b2b113024","Type":"ContainerDied","Data":"4de31d38d5970e8ab7991252cd1146b339371b3b1d89ed9ae5de1f87023938c7"}
Oct 30 20:20:54.369728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:54.369563    5002 scope.go:122] "RemoveContainer" containerID="4de31d38d5970e8ab7991252cd1146b339371b3b1d89ed9ae5de1f87023938c7"
Oct 30 20:20:55.375760 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:55.375723    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l4zm7" event={"ID":"70698df1-1b21-49a3-9cb1-131b2b113024","Type":"ContainerStarted","Data":"62b83fd5f4f9b642c402bf3054ba1b563cafc0f346f405c11c3016db9123111a"}
Oct 30 20:20:58.576861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:58.576816    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:20:58.877526 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:20:58.877436    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:00.402444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:00.402402    5002 generic.go:356] "Generic (PLEG): container finished" podID="ceacc798-3509-453a-b3ad-48abc179967b" containerID="855403074bd9b34a97d983347bc51a9f339502a475f13f6fadef37ce9e3b4cde" exitCode=0
Oct 30 20:21:00.403030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:00.402474    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn" event={"ID":"ceacc798-3509-453a-b3ad-48abc179967b","Type":"ContainerDied","Data":"855403074bd9b34a97d983347bc51a9f339502a475f13f6fadef37ce9e3b4cde"}
Oct 30 20:21:00.403030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:00.402896    5002 scope.go:122] "RemoveContainer" containerID="855403074bd9b34a97d983347bc51a9f339502a475f13f6fadef37ce9e3b4cde"
Oct 30 20:21:01.191650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:01.191613    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:01.367584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:01.367550    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:01.408236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:01.408205    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mxfmn" event={"ID":"ceacc798-3509-453a-b3ad-48abc179967b","Type":"ContainerStarted","Data":"ff843443f76ce89e4a6820c2cd62e69db13055f004902da9c062b48654310af1"}
Oct 30 20:21:07.113602 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:21:07.113570    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:21:09.703453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:09.703418    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:09.888599 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:09.888565    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:10.086340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:10.086312    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:10.267478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:10.267451    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:23.531291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:23.531259    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:24.645845 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:24.645808    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:25.394447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:21:25.394418    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:21:37.116643 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:21:37.116591    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:22:06.397452 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:06.397375    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:07.116198 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:22:07.116148    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:22:07.830551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:07.830497    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:08.003017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:08.002980    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:08.222155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:08.222125    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:08.398963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:08.398932    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:18.984085 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:18.984045    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:19.171722 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:19.171655    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:25.037525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:25.037491    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:31.951430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:31.951379    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:32.122585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:22:32.122558    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:22:37.114436 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:22:37.114403    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:23:07.113695 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:23:07.113660    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:23:20.075472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:23:20.075438    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:23:20.248292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:23:20.248263    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:23:25.318702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:23:25.318665    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:23:37.112908 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:23:37.112863    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:24:07.112992 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:24:07.112952    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:24:13.456163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:13.456125    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:14.954520 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:14.954490    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:15.997811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:15.997783    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:16.412341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.412279    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"]
Oct 30 20:24:16.412594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.412582    5002 cpu_manager.go:414] "containerMap: removing container" podUID="66fe23b4-c77d-49af-95de-6e87cc4eb223" containerName="registry"
Oct 30 20:24:16.412641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.412596    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fe23b4-c77d-49af-95de-6e87cc4eb223" containerName="registry"
Oct 30 20:24:16.412641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.412606    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry"
Oct 30 20:24:16.412641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.412610    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry"
Oct 30 20:24:16.412731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.412656    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fe23b4-c77d-49af-95de-6e87cc4eb223" containerName="registry"
Oct 30 20:24:16.412731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.412664    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d797780-22b8-4dee-bb0c-e2b0a38cceac" containerName="registry"
Oct 30 20:24:16.415714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.415700    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.475056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.475032    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.475185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.475072    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt7jm\" (UniqueName: \"kubernetes.io/projected/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-kube-api-access-tt7jm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.475185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.475169    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.522215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.522193    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 20:24:16.522215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.522196    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-9rjqm\""
Oct 30 20:24:16.522347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.522197    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 20:24:16.523048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.523031    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"]
Oct 30 20:24:16.575785 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.575767    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.575872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.575808    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.575872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.575826    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7jm\" (UniqueName: \"kubernetes.io/projected/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-kube-api-access-tt7jm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.576189 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.576168    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.576227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.576177    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.633455 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.633429    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7jm\" (UniqueName: \"kubernetes.io/projected/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-kube-api-access-tt7jm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:16.730537 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:16.730515    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:17.100221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:17.100160    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"]
Oct 30 20:24:17.102396 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:17.102371    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc2d89c_7ecd_4f41_9d8c_ebb6e47e3e8e.slice/crio-8c602f44bf2f4c3fe73ef5044066990eedfcc4c921ee3a111e296c32b607a845 WatchSource:0}: Error finding container 8c602f44bf2f4c3fe73ef5044066990eedfcc4c921ee3a111e296c32b607a845: Status 404 returned error can't find the container with id 8c602f44bf2f4c3fe73ef5044066990eedfcc4c921ee3a111e296c32b607a845
Oct 30 20:24:18.089206 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.089171    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk" event={"ID":"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e","Type":"ContainerStarted","Data":"8c602f44bf2f4c3fe73ef5044066990eedfcc4c921ee3a111e296c32b607a845"}
Oct 30 20:24:18.157620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.157596    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"]
Oct 30 20:24:18.160919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.160903    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.187399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.187378    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqq6\" (UniqueName: \"kubernetes.io/projected/04026416-aa2c-4595-bf79-22d1ee78ea2c-kube-api-access-8rqq6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.187488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.187410    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.187488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.187428    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.264322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.264296    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-6h5kd\""
Oct 30 20:24:18.265334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.265313    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"]
Oct 30 20:24:18.266138 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.266120    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"]
Oct 30 20:24:18.275325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.275307    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.288651 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.288626    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqq6\" (UniqueName: \"kubernetes.io/projected/04026416-aa2c-4595-bf79-22d1ee78ea2c-kube-api-access-8rqq6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.288782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.288680    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.288782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.288709    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.289152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.289134    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.289214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.289152    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.318691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.318669    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"]
Oct 30 20:24:18.328623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.328602    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-gfrpt\""
Oct 30 20:24:18.346418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.346362    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqq6\" (UniqueName: \"kubernetes.io/projected/04026416-aa2c-4595-bf79-22d1ee78ea2c-kube-api-access-8rqq6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.358211 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.358190    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"]
Oct 30 20:24:18.361813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.361793    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.389313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.389289    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.389416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.389349    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.389465 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.389444    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2xw\" (UniqueName: \"kubernetes.io/projected/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-kube-api-access-jn2xw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.389553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.389517    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c54q\" (UniqueName: \"kubernetes.io/projected/32b15d19-4f0c-4c40-8424-335e49fabb66-kube-api-access-8c54q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.389613 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.389598    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.389657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.389623    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.414846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.414821    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dtgs4\""
Oct 30 20:24:18.418994 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.418974    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"]
Oct 30 20:24:18.460283 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.458724    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"]
Oct 30 20:24:18.462382 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.462361    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.476209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.476190    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:18.490941 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.490917    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.491055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.490955    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2xw\" (UniqueName: \"kubernetes.io/projected/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-kube-api-access-jn2xw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.491055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.490990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c54q\" (UniqueName: \"kubernetes.io/projected/32b15d19-4f0c-4c40-8424-335e49fabb66-kube-api-access-8c54q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.491055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491017    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.491055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491045    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.491268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491075    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt9bk\" (UniqueName: \"kubernetes.io/projected/89ac82d8-3339-4323-895e-40c552bd5e9d-kube-api-access-kt9bk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.491268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491133    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.491268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491166    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.491268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491191    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.492277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491673    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.492277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491802    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.492277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.491811    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.492277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.492168    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.515525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.515496    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-wh5zz\""
Oct 30 20:24:18.520873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.520849    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"]
Oct 30 20:24:18.549518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.549144    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2xw\" (UniqueName: \"kubernetes.io/projected/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-kube-api-access-jn2xw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.549663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.549565    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c54q\" (UniqueName: \"kubernetes.io/projected/32b15d19-4f0c-4c40-8424-335e49fabb66-kube-api-access-8c54q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.591655 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.591628    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:18.591853 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.591823    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.591985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.591875    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.591985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.591946    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt9bk\" (UniqueName: \"kubernetes.io/projected/89ac82d8-3339-4323-895e-40c552bd5e9d-kube-api-access-kt9bk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.592236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.592213    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.592283 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.592253    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.649668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.649639    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt9bk\" (UniqueName: \"kubernetes.io/projected/89ac82d8-3339-4323-895e-40c552bd5e9d-kube-api-access-kt9bk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.680877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.680852    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:18.779606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.779583    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:18.859078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.859056    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"]
Oct 30 20:24:18.861113 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:18.861082    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04026416_aa2c_4595_bf79_22d1ee78ea2c.slice/crio-2207b005714c9f04bfc1b4a8282a747823f9f3f67964b2f416073ee3f138cad1 WatchSource:0}: Error finding container 2207b005714c9f04bfc1b4a8282a747823f9f3f67964b2f416073ee3f138cad1: Status 404 returned error can't find the container with id 2207b005714c9f04bfc1b4a8282a747823f9f3f67964b2f416073ee3f138cad1
Oct 30 20:24:18.965282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:18.965257    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"]
Oct 30 20:24:18.968265 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:18.968240    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ccabc5_fd7d_4358_b4d7_55124d94baa3.slice/crio-fabc6d2f6899d150dad30f946152d6cdf175b8a9e27104a2087af28f5c8ea48e WatchSource:0}: Error finding container fabc6d2f6899d150dad30f946152d6cdf175b8a9e27104a2087af28f5c8ea48e: Status 404 returned error can't find the container with id fabc6d2f6899d150dad30f946152d6cdf175b8a9e27104a2087af28f5c8ea48e
Oct 30 20:24:19.050321 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:19.050296    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"]
Oct 30 20:24:19.054087 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:19.054050    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b15d19_4f0c_4c40_8424_335e49fabb66.slice/crio-bb2b77c0b7e7db714fd96a40d66d6918cc9e96ff068ea44dc9cc480c9dedccd2 WatchSource:0}: Error finding container bb2b77c0b7e7db714fd96a40d66d6918cc9e96ff068ea44dc9cc480c9dedccd2: Status 404 returned error can't find the container with id bb2b77c0b7e7db714fd96a40d66d6918cc9e96ff068ea44dc9cc480c9dedccd2
Oct 30 20:24:19.095056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:19.095027    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd" event={"ID":"32b15d19-4f0c-4c40-8424-335e49fabb66","Type":"ContainerStarted","Data":"bb2b77c0b7e7db714fd96a40d66d6918cc9e96ff068ea44dc9cc480c9dedccd2"}
Oct 30 20:24:19.096290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:19.096262    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82" event={"ID":"a1ccabc5-fd7d-4358-b4d7-55124d94baa3","Type":"ContainerStarted","Data":"fabc6d2f6899d150dad30f946152d6cdf175b8a9e27104a2087af28f5c8ea48e"}
Oct 30 20:24:19.097473 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:19.097449    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s" event={"ID":"04026416-aa2c-4595-bf79-22d1ee78ea2c","Type":"ContainerStarted","Data":"2207b005714c9f04bfc1b4a8282a747823f9f3f67964b2f416073ee3f138cad1"}
Oct 30 20:24:19.153278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:19.153257    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"]
Oct 30 20:24:19.155229 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:19.155208    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ac82d8_3339_4323_895e_40c552bd5e9d.slice/crio-54d3325afe41238880a129b56607e4a397ff4c7393eadad44b595a4c55cdc845 WatchSource:0}: Error finding container 54d3325afe41238880a129b56607e4a397ff4c7393eadad44b595a4c55cdc845: Status 404 returned error can't find the container with id 54d3325afe41238880a129b56607e4a397ff4c7393eadad44b595a4c55cdc845
Oct 30 20:24:20.105164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.104001    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h" event={"ID":"89ac82d8-3339-4323-895e-40c552bd5e9d","Type":"ContainerStarted","Data":"54d3325afe41238880a129b56607e4a397ff4c7393eadad44b595a4c55cdc845"}
Oct 30 20:24:20.185209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.185177    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"]
Oct 30 20:24:20.189385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.189363    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.246942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.246909    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"]
Oct 30 20:24:20.295792 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.295755    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rbgp6\""
Oct 30 20:24:20.304313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.304039    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.304313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.304115    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2hw\" (UniqueName: \"kubernetes.io/projected/3c1187a4-aba6-4d72-85ac-9e464a1c688a-kube-api-access-kk2hw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.304313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.304205    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.406991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.405322    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.406991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.405365    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2hw\" (UniqueName: \"kubernetes.io/projected/3c1187a4-aba6-4d72-85ac-9e464a1c688a-kube-api-access-kk2hw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.406991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.405425    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.406991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.406092    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.406991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.406481    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.472360 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.472292    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2hw\" (UniqueName: \"kubernetes.io/projected/3c1187a4-aba6-4d72-85ac-9e464a1c688a-kube-api-access-kk2hw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.484569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.483688    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"]
Oct 30 20:24:20.488525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.488503    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.508711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.508684    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:20.546947 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.546920    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-2rksq\""
Oct 30 20:24:20.548373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.548347    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"]
Oct 30 20:24:20.608155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.608124    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.608264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.608192    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.608264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.608222    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77f25\" (UniqueName: \"kubernetes.io/projected/a6425949-c8d7-415a-83f5-eb52f5ea3463-kube-api-access-77f25\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.717370 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.717336    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.717576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.717379    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77f25\" (UniqueName: \"kubernetes.io/projected/a6425949-c8d7-415a-83f5-eb52f5ea3463-kube-api-access-77f25\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.717576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.717470    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.717844 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.717824    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.717844 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.717832    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.778080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.778019    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77f25\" (UniqueName: \"kubernetes.io/projected/a6425949-c8d7-415a-83f5-eb52f5ea3463-kube-api-access-77f25\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.812866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.812835    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:20.902356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.899521    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:20.921142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:20.921081    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"]
Oct 30 20:24:20.927941 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:20.927908    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1187a4_aba6_4d72_85ac_9e464a1c688a.slice/crio-f1d661cdef497e5abcbb4d198dc8658e3e54c2cf34e00a6bb16ef38de38106e6 WatchSource:0}: Error finding container f1d661cdef497e5abcbb4d198dc8658e3e54c2cf34e00a6bb16ef38de38106e6: Status 404 returned error can't find the container with id f1d661cdef497e5abcbb4d198dc8658e3e54c2cf34e00a6bb16ef38de38106e6
Oct 30 20:24:21.079130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.079053    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:21.114960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.114921    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7" event={"ID":"3c1187a4-aba6-4d72-85ac-9e464a1c688a","Type":"ContainerStarted","Data":"f1d661cdef497e5abcbb4d198dc8658e3e54c2cf34e00a6bb16ef38de38106e6"}
Oct 30 20:24:21.208974 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.208944    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"]
Oct 30 20:24:21.210761 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:21.210732    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6425949_c8d7_415a_83f5_eb52f5ea3463.slice/crio-f07c61b9e5174fa98931375533cff3886468ac1adbb1f51fdd7d9293af564c9a WatchSource:0}: Error finding container f07c61b9e5174fa98931375533cff3886468ac1adbb1f51fdd7d9293af564c9a: Status 404 returned error can't find the container with id f07c61b9e5174fa98931375533cff3886468ac1adbb1f51fdd7d9293af564c9a
Oct 30 20:24:21.278489 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.278451    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"]
Oct 30 20:24:21.283204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.283181    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.326592 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326558    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.326745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326610    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.326745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326643    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.326745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326665    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.326745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326694    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.326745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326730    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.327156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326753    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.327156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326810    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkpz\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-kube-api-access-8lkpz\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.327156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.326877    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.336717 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.336654    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:24:21.336852 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.336779    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7k2d5\""
Oct 30 20:24:21.344001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.343978    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"]
Oct 30 20:24:21.408544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.408500    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:21.428183 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428156    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428208    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428261    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428290    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428315    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428344    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428377    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428397    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.428488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428438    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkpz\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-kube-api-access-8lkpz\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.429343 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.428971    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.430018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.429589    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.430018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.429885    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.430319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.430260    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.430457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.430425    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.433553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.433506    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.440202 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.440180    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.489889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.489864    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkpz\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-kube-api-access-8lkpz\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.491943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.491916    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fmmz4\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.581966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.581928    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"]
Oct 30 20:24:21.586984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.586927    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.601430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.600847    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:21.611273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.608576    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:21.630683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630647    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630705    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630732    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630763    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630807    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630838    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4c9\" (UniqueName: \"kubernetes.io/projected/6b88f899-73f5-4c2f-8af5-9f655fbd6238-kube-api-access-vf4c9\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630872    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630921    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.630966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.630957    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.648912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.648875    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"]
Oct 30 20:24:21.658678 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.658657    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-gdfhq\""
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732486    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732552    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732579    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732608    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732647    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732676    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4c9\" (UniqueName: \"kubernetes.io/projected/6b88f899-73f5-4c2f-8af5-9f655fbd6238-kube-api-access-vf4c9\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732709    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732758    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.732797    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.733949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.733557    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.736197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.734863    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.736197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.735385    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.736197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.735641    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.736197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.736159    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.739650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.739617    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.743086 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.743060    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.787051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.787019    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:21.793166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.793143    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4c9\" (UniqueName: \"kubernetes.io/projected/6b88f899-73f5-4c2f-8af5-9f655fbd6238-kube-api-access-vf4c9\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.797136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.797097    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b88f899-73f5-4c2f-8af5-9f655fbd6238-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl\" (UID: \"6b88f899-73f5-4c2f-8af5-9f655fbd6238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:21.913594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:21.913491    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:22.030508 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:22.029560    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"]
Oct 30 20:24:22.040845 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:24:22.040779    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d89af5_52dd_4707_ab01_dd7b28381b9a.slice/crio-134ccf037c25784fe0dfc435fc77adba0dec9f94969753a937525fb626a82c5a WatchSource:0}: Error finding container 134ccf037c25784fe0dfc435fc77adba0dec9f94969753a937525fb626a82c5a: Status 404 returned error can't find the container with id 134ccf037c25784fe0dfc435fc77adba0dec9f94969753a937525fb626a82c5a
Oct 30 20:24:22.123820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:22.123728    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" event={"ID":"f1d89af5-52dd-4707-ab01-dd7b28381b9a","Type":"ContainerStarted","Data":"134ccf037c25784fe0dfc435fc77adba0dec9f94969753a937525fb626a82c5a"}
Oct 30 20:24:22.129264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:22.129203    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll" event={"ID":"a6425949-c8d7-415a-83f5-eb52f5ea3463","Type":"ContainerStarted","Data":"f07c61b9e5174fa98931375533cff3886468ac1adbb1f51fdd7d9293af564c9a"}
Oct 30 20:24:22.332750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:22.332629    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"]
Oct 30 20:24:22.641777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:22.641656    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:23.147826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:23.147785    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl" event={"ID":"6b88f899-73f5-4c2f-8af5-9f655fbd6238","Type":"ContainerStarted","Data":"864a80be62d0499909e3266f0eaab73ece91cbea88d6c5d419962f6120803e09"}
Oct 30 20:24:25.548788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:25.548751    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:25.726045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:25.725974    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:24:30.719407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:30.719368    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:30.719813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:30.719479    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:30.719813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:30.719508    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:31.210882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:31.210181    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" event={"ID":"f1d89af5-52dd-4707-ab01-dd7b28381b9a","Type":"ContainerStarted","Data":"7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee"}
Oct 30 20:24:31.430798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:31.430737    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podStartSLOduration=1.757967819 podStartE2EDuration="10.430722356s" totalImagesPullingTime="8.672754537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:31.379924032 +0000 UTC m=+297.528975020" watchObservedRunningTime="2026-10-30 20:24:31.430722356 +0000 UTC m=+297.579773340"
Oct 30 20:24:31.603415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:31.603335    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:31.609698 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:31.609675    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:32.216912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:32.216831    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:32.218092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:32.218065    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:24:33.225780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:33.225742    5002 generic.go:356] "Generic (PLEG): container finished" podID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerID="0827c3e6cab2312d5fac50b5167b0e023729c24a52c6c8ae057f90bf26912238" exitCode=0
Oct 30 20:24:33.226280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:33.225823    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk" event={"ID":"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e","Type":"ContainerDied","Data":"0827c3e6cab2312d5fac50b5167b0e023729c24a52c6c8ae057f90bf26912238"}
Oct 30 20:24:33.729200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:33.729155    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:33.729324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:33.729250    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:33.729324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:33.729296    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:33.990526 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:33.990499    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:24:33.992520 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:33.992500    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:24:34.233493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.232389    5002 generic.go:356] "Generic (PLEG): container finished" podID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerID="c07f594e8c66d1456f12182af4052bd0cdf6cfce6792f738ceb18b612885fa96" exitCode=0
Oct 30 20:24:34.233493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.232475    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82" event={"ID":"a1ccabc5-fd7d-4358-b4d7-55124d94baa3","Type":"ContainerDied","Data":"c07f594e8c66d1456f12182af4052bd0cdf6cfce6792f738ceb18b612885fa96"}
Oct 30 20:24:34.236035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.236005    5002 generic.go:356] "Generic (PLEG): container finished" podID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerID="67e0e9fd5dea50cf31c6cd3e06dd2e425bc50e60a97fe6ca5d0b568509367457" exitCode=0
Oct 30 20:24:34.236145 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.236125    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s" event={"ID":"04026416-aa2c-4595-bf79-22d1ee78ea2c","Type":"ContainerDied","Data":"67e0e9fd5dea50cf31c6cd3e06dd2e425bc50e60a97fe6ca5d0b568509367457"}
Oct 30 20:24:34.238706 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.238673    5002 generic.go:356] "Generic (PLEG): container finished" podID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerID="416acefa1a3774a0c6cc2aefdffb57ccc98d47dd2bbcd795b249289a918e8d55" exitCode=0
Oct 30 20:24:34.238870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.238798    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd" event={"ID":"32b15d19-4f0c-4c40-8424-335e49fabb66","Type":"ContainerDied","Data":"416acefa1a3774a0c6cc2aefdffb57ccc98d47dd2bbcd795b249289a918e8d55"}
Oct 30 20:24:34.242071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.242047    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl" event={"ID":"6b88f899-73f5-4c2f-8af5-9f655fbd6238","Type":"ContainerStarted","Data":"9a7344f18d55f02c3befb9ce8f0be9866702ab53c3053b20e9330c907b21ad66"}
Oct 30 20:24:34.913841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.913723    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:34.919506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:34.919265    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:35.041447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:35.041389    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl" podStartSLOduration=2.653184328 podStartE2EDuration="14.041372344s" totalImagesPullingTime="11.388188016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:34.702381303 +0000 UTC m=+300.851432290" watchObservedRunningTime="2026-10-30 20:24:35.041372344 +0000 UTC m=+301.190423325"
Oct 30 20:24:35.250051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:35.248871    5002 generic.go:356] "Generic (PLEG): container finished" podID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerID="0a40af151cae07e3edfdeb7fdea0ed8f6f5ee7b7a0105e605d4b38dea2abdf76" exitCode=0
Oct 30 20:24:35.250051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:35.249956    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h" event={"ID":"89ac82d8-3339-4323-895e-40c552bd5e9d","Type":"ContainerDied","Data":"0a40af151cae07e3edfdeb7fdea0ed8f6f5ee7b7a0105e605d4b38dea2abdf76"}
Oct 30 20:24:35.250051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:35.250012    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:35.251200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:35.250759    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl"
Oct 30 20:24:36.257107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.257081    5002 generic.go:356] "Generic (PLEG): container finished" podID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerID="d1aa34bc7eacfd7748d68644d4cc04941808f89b39185de0ccf2b60fea18a1b2" exitCode=0
Oct 30 20:24:36.257546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.257160    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s" event={"ID":"04026416-aa2c-4595-bf79-22d1ee78ea2c","Type":"ContainerDied","Data":"d1aa34bc7eacfd7748d68644d4cc04941808f89b39185de0ccf2b60fea18a1b2"}
Oct 30 20:24:36.259936 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.259915    5002 generic.go:356] "Generic (PLEG): container finished" podID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerID="28333e27b0c42ac6c09a07f01cb6ae7fc65434aec8a99c16949908229305c9c4" exitCode=0
Oct 30 20:24:36.260053 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.260032    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd" event={"ID":"32b15d19-4f0c-4c40-8424-335e49fabb66","Type":"ContainerDied","Data":"28333e27b0c42ac6c09a07f01cb6ae7fc65434aec8a99c16949908229305c9c4"}
Oct 30 20:24:36.262612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.262583    5002 generic.go:356] "Generic (PLEG): container finished" podID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerID="5b6d9179d2af063128a4c26fa6fed9e2963a84ac2a886cfac43b4b0411dc1e5c" exitCode=0
Oct 30 20:24:36.262922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.262901    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll" event={"ID":"a6425949-c8d7-415a-83f5-eb52f5ea3463","Type":"ContainerDied","Data":"5b6d9179d2af063128a4c26fa6fed9e2963a84ac2a886cfac43b4b0411dc1e5c"}
Oct 30 20:24:36.264389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.264365    5002 generic.go:356] "Generic (PLEG): container finished" podID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerID="3f735ecf11329933873cbbd39babe10b990632ea8a936a49765b50acf06cff7e" exitCode=0
Oct 30 20:24:36.264468 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:36.264432    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7" event={"ID":"3c1187a4-aba6-4d72-85ac-9e464a1c688a","Type":"ContainerDied","Data":"3f735ecf11329933873cbbd39babe10b990632ea8a936a49765b50acf06cff7e"}
Oct 30 20:24:37.123281 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:24:37.123246    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:24:37.271616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:37.271589    5002 generic.go:356] "Generic (PLEG): container finished" podID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerID="19027ae84a4dd515c610d63ca392c70476133e71e1ff37fe66183facf7bb8b4f" exitCode=0
Oct 30 20:24:37.272030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:37.271687    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82" event={"ID":"a1ccabc5-fd7d-4358-b4d7-55124d94baa3","Type":"ContainerDied","Data":"19027ae84a4dd515c610d63ca392c70476133e71e1ff37fe66183facf7bb8b4f"}
Oct 30 20:24:37.274597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:37.274572    5002 generic.go:356] "Generic (PLEG): container finished" podID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerID="6ae28d1e3f685a0022630df7e590e3bcf2982ae35f52ce9bc300744dbd887303" exitCode=0
Oct 30 20:24:37.274744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:37.274642    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h" event={"ID":"89ac82d8-3339-4323-895e-40c552bd5e9d","Type":"ContainerDied","Data":"6ae28d1e3f685a0022630df7e590e3bcf2982ae35f52ce9bc300744dbd887303"}
Oct 30 20:24:37.277160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:37.277138    5002 generic.go:356] "Generic (PLEG): container finished" podID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerID="3f44f920da1468def99607614483ec240a33ea085b065d309fc911219c328479" exitCode=0
Oct 30 20:24:37.277249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:37.277163    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll" event={"ID":"a6425949-c8d7-415a-83f5-eb52f5ea3463","Type":"ContainerDied","Data":"3f44f920da1468def99607614483ec240a33ea085b065d309fc911219c328479"}
Oct 30 20:24:38.287366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:38.286380    5002 generic.go:356] "Generic (PLEG): container finished" podID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerID="2a7049e4ff0cbf85a732d85612aabc4ae0962e5600a514d1f4077e24edf4b19f" exitCode=0
Oct 30 20:24:38.287366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:38.286483    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk" event={"ID":"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e","Type":"ContainerDied","Data":"2a7049e4ff0cbf85a732d85612aabc4ae0962e5600a514d1f4077e24edf4b19f"}
Oct 30 20:24:42.336404 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:42.336359    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7" event={"ID":"3c1187a4-aba6-4d72-85ac-9e464a1c688a","Type":"ContainerStarted","Data":"72565e2c69992c6f7a81d4224a3338368f777c73ab7ab31d175ccf8a72106e6c"}
Oct 30 20:24:43.354256 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:43.354217    5002 generic.go:356] "Generic (PLEG): container finished" podID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerID="72565e2c69992c6f7a81d4224a3338368f777c73ab7ab31d175ccf8a72106e6c" exitCode=0
Oct 30 20:24:43.354752 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:43.354323    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7" event={"ID":"3c1187a4-aba6-4d72-85ac-9e464a1c688a","Type":"ContainerDied","Data":"72565e2c69992c6f7a81d4224a3338368f777c73ab7ab31d175ccf8a72106e6c"}
Oct 30 20:24:43.357560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:43.357517    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:24:50.420075 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:50.420049    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd" event={"ID":"32b15d19-4f0c-4c40-8424-335e49fabb66","Type":"ContainerStarted","Data":"bbd14ef540dcd5128e6c8c77aaa7c9f39ff9ee71b54996288b2171eef24ae57f"}
Oct 30 20:24:50.541238 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:50.541176    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd" podStartSLOduration=1.958900984 podStartE2EDuration="32.541156466s" totalImagesPullingTime="30.582255482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:50.539424902 +0000 UTC m=+316.688475889" watchObservedRunningTime="2026-10-30 20:24:50.541156466 +0000 UTC m=+316.690207455"
Oct 30 20:24:51.431017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:51.430595    5002 generic.go:356] "Generic (PLEG): container finished" podID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerID="51221dbbaad7f39bd242bff3518ab20427814650772a1f1178215682122ed7f4" exitCode=0
Oct 30 20:24:51.431017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:51.430690    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s" event={"ID":"04026416-aa2c-4595-bf79-22d1ee78ea2c","Type":"ContainerDied","Data":"51221dbbaad7f39bd242bff3518ab20427814650772a1f1178215682122ed7f4"}
Oct 30 20:24:51.435307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:51.435285    5002 generic.go:356] "Generic (PLEG): container finished" podID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerID="bbd14ef540dcd5128e6c8c77aaa7c9f39ff9ee71b54996288b2171eef24ae57f" exitCode=0
Oct 30 20:24:51.435406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:51.435392    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd" event={"ID":"32b15d19-4f0c-4c40-8424-335e49fabb66","Type":"ContainerDied","Data":"bbd14ef540dcd5128e6c8c77aaa7c9f39ff9ee71b54996288b2171eef24ae57f"}
Oct 30 20:24:52.441921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.441894    5002 generic.go:356] "Generic (PLEG): container finished" podID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerID="470010e57ef779f75d0cccebb2714073d671d8214408078dc6e730b6872d5b08" exitCode=0
Oct 30 20:24:52.442297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.441968    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h" event={"ID":"89ac82d8-3339-4323-895e-40c552bd5e9d","Type":"ContainerDied","Data":"470010e57ef779f75d0cccebb2714073d671d8214408078dc6e730b6872d5b08"}
Oct 30 20:24:52.443984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.443965    5002 generic.go:356] "Generic (PLEG): container finished" podID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerID="2e2f56a8b33c10bab08cb81c5bc2e3ac4ec2280279292c60814604686bc3c693" exitCode=0
Oct 30 20:24:52.444087 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.444040    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll" event={"ID":"a6425949-c8d7-415a-83f5-eb52f5ea3463","Type":"ContainerDied","Data":"2e2f56a8b33c10bab08cb81c5bc2e3ac4ec2280279292c60814604686bc3c693"}
Oct 30 20:24:52.445850 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.445833    5002 generic.go:356] "Generic (PLEG): container finished" podID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerID="3643ad82d6a8b8140fb93e35d9879cf0573411f913c4394d60fe20d3c4c978c5" exitCode=0
Oct 30 20:24:52.445937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.445914    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7" event={"ID":"3c1187a4-aba6-4d72-85ac-9e464a1c688a","Type":"ContainerDied","Data":"3643ad82d6a8b8140fb93e35d9879cf0573411f913c4394d60fe20d3c4c978c5"}
Oct 30 20:24:52.447845 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.447829    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk" event={"ID":"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e","Type":"ContainerStarted","Data":"3b4423401aa097760836784af62d8d2628970035245c56d7c6292800281d47bd"}
Oct 30 20:24:52.449676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.449657    5002 generic.go:356] "Generic (PLEG): container finished" podID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerID="1c2c36b32bb2e28657d6defe4a03ca3b3dd6aa924e681a5f21aace1f6bd3da98" exitCode=0
Oct 30 20:24:52.449772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.449738    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82" event={"ID":"a1ccabc5-fd7d-4358-b4d7-55124d94baa3","Type":"ContainerDied","Data":"1c2c36b32bb2e28657d6defe4a03ca3b3dd6aa924e681a5f21aace1f6bd3da98"}
Oct 30 20:24:52.576652 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.576631    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:52.601424 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.601405    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:52.740410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.740389    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-bundle\") pod \"04026416-aa2c-4595-bf79-22d1ee78ea2c\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") "
Oct 30 20:24:52.740507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.740429    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04026416-aa2c-4595-bf79-22d1ee78ea2c-kube-api-access-8rqq6\" (UniqueName: \"kubernetes.io/projected/04026416-aa2c-4595-bf79-22d1ee78ea2c-kube-api-access-8rqq6\") pod \"04026416-aa2c-4595-bf79-22d1ee78ea2c\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") "
Oct 30 20:24:52.740507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.740471    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b15d19-4f0c-4c40-8424-335e49fabb66-kube-api-access-8c54q\" (UniqueName: \"kubernetes.io/projected/32b15d19-4f0c-4c40-8424-335e49fabb66-kube-api-access-8c54q\") pod \"32b15d19-4f0c-4c40-8424-335e49fabb66\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") "
Oct 30 20:24:52.740624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.740508    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-bundle\") pod \"32b15d19-4f0c-4c40-8424-335e49fabb66\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") "
Oct 30 20:24:52.740624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.740524    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-util\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-util\") pod \"32b15d19-4f0c-4c40-8424-335e49fabb66\" (UID: \"32b15d19-4f0c-4c40-8424-335e49fabb66\") "
Oct 30 20:24:52.740624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.740576    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-util\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-util\") pod \"04026416-aa2c-4595-bf79-22d1ee78ea2c\" (UID: \"04026416-aa2c-4595-bf79-22d1ee78ea2c\") "
Oct 30 20:24:52.741115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.741087    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-bundle" pod "32b15d19-4f0c-4c40-8424-335e49fabb66" (UID: "32b15d19-4f0c-4c40-8424-335e49fabb66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:52.741115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.741101    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-bundle" pod "04026416-aa2c-4595-bf79-22d1ee78ea2c" (UID: "04026416-aa2c-4595-bf79-22d1ee78ea2c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:52.742909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.742887    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04026416-aa2c-4595-bf79-22d1ee78ea2c-kube-api-access-8rqq6" pod "04026416-aa2c-4595-bf79-22d1ee78ea2c" (UID: "04026416-aa2c-4595-bf79-22d1ee78ea2c"). InnerVolumeSpecName "kube-api-access-8rqq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:52.743008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.742989    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b15d19-4f0c-4c40-8424-335e49fabb66-kube-api-access-8c54q" pod "32b15d19-4f0c-4c40-8424-335e49fabb66" (UID: "32b15d19-4f0c-4c40-8424-335e49fabb66"). InnerVolumeSpecName "kube-api-access-8c54q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:52.746610 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.746587    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-util" pod "32b15d19-4f0c-4c40-8424-335e49fabb66" (UID: "32b15d19-4f0c-4c40-8424-335e49fabb66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:52.747376 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.747352    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-util" pod "04026416-aa2c-4595-bf79-22d1ee78ea2c" (UID: "04026416-aa2c-4595-bf79-22d1ee78ea2c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:52.841546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.841488    5002 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.841546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.841508    5002 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b15d19-4f0c-4c40-8424-335e49fabb66-util\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.841546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.841516    5002 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-util\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.841546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.841523    5002 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04026416-aa2c-4595-bf79-22d1ee78ea2c-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.841546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.841544    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqq6\" (UniqueName: \"kubernetes.io/projected/04026416-aa2c-4595-bf79-22d1ee78ea2c-kube-api-access-8rqq6\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.841753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.841553    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c54q\" (UniqueName: \"kubernetes.io/projected/32b15d19-4f0c-4c40-8424-335e49fabb66-kube-api-access-8c54q\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.918682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:52.918642    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk" podStartSLOduration=3.756475925 podStartE2EDuration="36.918630927s" totalImagesPullingTime="33.162155002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:52.917715173 +0000 UTC m=+319.066766161" watchObservedRunningTime="2026-10-30 20:24:52.918630927 +0000 UTC m=+319.067681914"
Oct 30 20:24:53.455842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.455819    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd"
Oct 30 20:24:53.456257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.455844    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bszsd" event={"ID":"32b15d19-4f0c-4c40-8424-335e49fabb66","Type":"ContainerDied","Data":"bb2b77c0b7e7db714fd96a40d66d6918cc9e96ff068ea44dc9cc480c9dedccd2"}
Oct 30 20:24:53.456257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.455888    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb2b77c0b7e7db714fd96a40d66d6918cc9e96ff068ea44dc9cc480c9dedccd2"
Oct 30 20:24:53.458040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.458014    5002 generic.go:356] "Generic (PLEG): container finished" podID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerID="3b4423401aa097760836784af62d8d2628970035245c56d7c6292800281d47bd" exitCode=0
Oct 30 20:24:53.458180 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.458077    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk" event={"ID":"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e","Type":"ContainerDied","Data":"3b4423401aa097760836784af62d8d2628970035245c56d7c6292800281d47bd"}
Oct 30 20:24:53.459890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.459826    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s" event={"ID":"04026416-aa2c-4595-bf79-22d1ee78ea2c","Type":"ContainerDied","Data":"2207b005714c9f04bfc1b4a8282a747823f9f3f67964b2f416073ee3f138cad1"}
Oct 30 20:24:53.459890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.459852    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clln7s"
Oct 30 20:24:53.459890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.459869    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2207b005714c9f04bfc1b4a8282a747823f9f3f67964b2f416073ee3f138cad1"
Oct 30 20:24:53.596229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.596207    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:53.638250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.638228    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:53.653854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.653835    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:53.661221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.661202    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:53.748836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.748811    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-kube-api-access-jn2xw\" (UniqueName: \"kubernetes.io/projected/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-kube-api-access-jn2xw\") pod \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") "
Oct 30 20:24:53.748984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.748857    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-util\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-util\") pod \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") "
Oct 30 20:24:53.748984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.748899    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ac82d8-3339-4323-895e-40c552bd5e9d-kube-api-access-kt9bk\" (UniqueName: \"kubernetes.io/projected/89ac82d8-3339-4323-895e-40c552bd5e9d-kube-api-access-kt9bk\") pod \"89ac82d8-3339-4323-895e-40c552bd5e9d\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") "
Oct 30 20:24:53.748984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.748931    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1187a4-aba6-4d72-85ac-9e464a1c688a-kube-api-access-kk2hw\" (UniqueName: \"kubernetes.io/projected/3c1187a4-aba6-4d72-85ac-9e464a1c688a-kube-api-access-kk2hw\") pod \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") "
Oct 30 20:24:53.748984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.748961    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-util\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-util\") pod \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") "
Oct 30 20:24:53.749186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.748985    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-bundle\") pod \"a6425949-c8d7-415a-83f5-eb52f5ea3463\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") "
Oct 30 20:24:53.749186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749030    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-util\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-util\") pod \"a6425949-c8d7-415a-83f5-eb52f5ea3463\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") "
Oct 30 20:24:53.749186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749051    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-bundle\") pod \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\" (UID: \"3c1187a4-aba6-4d72-85ac-9e464a1c688a\") "
Oct 30 20:24:53.749186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749083    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-bundle\") pod \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\" (UID: \"a1ccabc5-fd7d-4358-b4d7-55124d94baa3\") "
Oct 30 20:24:53.749186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749106    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-util\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-util\") pod \"89ac82d8-3339-4323-895e-40c552bd5e9d\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") "
Oct 30 20:24:53.749186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749138    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-bundle\") pod \"89ac82d8-3339-4323-895e-40c552bd5e9d\" (UID: \"89ac82d8-3339-4323-895e-40c552bd5e9d\") "
Oct 30 20:24:53.749186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749168    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6425949-c8d7-415a-83f5-eb52f5ea3463-kube-api-access-77f25\" (UniqueName: \"kubernetes.io/projected/a6425949-c8d7-415a-83f5-eb52f5ea3463-kube-api-access-77f25\") pod \"a6425949-c8d7-415a-83f5-eb52f5ea3463\" (UID: \"a6425949-c8d7-415a-83f5-eb52f5ea3463\") "
Oct 30 20:24:53.749514 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749444    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-bundle" pod "3c1187a4-aba6-4d72-85ac-9e464a1c688a" (UID: "3c1187a4-aba6-4d72-85ac-9e464a1c688a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.750076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.749959    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-bundle" pod "a6425949-c8d7-415a-83f5-eb52f5ea3463" (UID: "a6425949-c8d7-415a-83f5-eb52f5ea3463"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.751457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.751429    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ac82d8-3339-4323-895e-40c552bd5e9d-kube-api-access-kt9bk" pod "89ac82d8-3339-4323-895e-40c552bd5e9d" (UID: "89ac82d8-3339-4323-895e-40c552bd5e9d"). InnerVolumeSpecName "kube-api-access-kt9bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:53.751815 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.751696    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-kube-api-access-jn2xw" pod "a1ccabc5-fd7d-4358-b4d7-55124d94baa3" (UID: "a1ccabc5-fd7d-4358-b4d7-55124d94baa3"). InnerVolumeSpecName "kube-api-access-jn2xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:53.751953 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.751921    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-bundle" pod "a1ccabc5-fd7d-4358-b4d7-55124d94baa3" (UID: "a1ccabc5-fd7d-4358-b4d7-55124d94baa3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.752040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.751979    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6425949-c8d7-415a-83f5-eb52f5ea3463-kube-api-access-77f25" pod "a6425949-c8d7-415a-83f5-eb52f5ea3463" (UID: "a6425949-c8d7-415a-83f5-eb52f5ea3463"). InnerVolumeSpecName "kube-api-access-77f25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:53.752392 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.752362    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1187a4-aba6-4d72-85ac-9e464a1c688a-kube-api-access-kk2hw" pod "3c1187a4-aba6-4d72-85ac-9e464a1c688a" (UID: "3c1187a4-aba6-4d72-85ac-9e464a1c688a"). InnerVolumeSpecName "kube-api-access-kk2hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:53.752392 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.752382    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-bundle" pod "89ac82d8-3339-4323-895e-40c552bd5e9d" (UID: "89ac82d8-3339-4323-895e-40c552bd5e9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.755811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.755788    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-util" pod "a6425949-c8d7-415a-83f5-eb52f5ea3463" (UID: "a6425949-c8d7-415a-83f5-eb52f5ea3463"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.755968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.755944    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-util" pod "3c1187a4-aba6-4d72-85ac-9e464a1c688a" (UID: "3c1187a4-aba6-4d72-85ac-9e464a1c688a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.755968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.755944    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-util" pod "a1ccabc5-fd7d-4358-b4d7-55124d94baa3" (UID: "a1ccabc5-fd7d-4358-b4d7-55124d94baa3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.756475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.756457    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-util" pod "89ac82d8-3339-4323-895e-40c552bd5e9d" (UID: "89ac82d8-3339-4323-895e-40c552bd5e9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:53.850300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850275    5002 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-util\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850299    5002 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850308    5002 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6425949-c8d7-415a-83f5-eb52f5ea3463-util\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850316    5002 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850323    5002 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850333    5002 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-util\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850341    5002 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ac82d8-3339-4323-895e-40c552bd5e9d-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850348    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77f25\" (UniqueName: \"kubernetes.io/projected/a6425949-c8d7-415a-83f5-eb52f5ea3463-kube-api-access-77f25\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850361    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2xw\" (UniqueName: \"kubernetes.io/projected/a1ccabc5-fd7d-4358-b4d7-55124d94baa3-kube-api-access-jn2xw\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850373    5002 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c1187a4-aba6-4d72-85ac-9e464a1c688a-util\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850381    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt9bk\" (UniqueName: \"kubernetes.io/projected/89ac82d8-3339-4323-895e-40c552bd5e9d-kube-api-access-kt9bk\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.850423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:53.850389    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk2hw\" (UniqueName: \"kubernetes.io/projected/3c1187a4-aba6-4d72-85ac-9e464a1c688a-kube-api-access-kk2hw\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:54.466106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.466075    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h" event={"ID":"89ac82d8-3339-4323-895e-40c552bd5e9d","Type":"ContainerDied","Data":"54d3325afe41238880a129b56607e4a397ff4c7393eadad44b595a4c55cdc845"}
Oct 30 20:24:54.466497 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.466110    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9446h"
Oct 30 20:24:54.466497 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.466155    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d3325afe41238880a129b56607e4a397ff4c7393eadad44b595a4c55cdc845"
Oct 30 20:24:54.468095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.468071    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll" event={"ID":"a6425949-c8d7-415a-83f5-eb52f5ea3463","Type":"ContainerDied","Data":"f07c61b9e5174fa98931375533cff3886468ac1adbb1f51fdd7d9293af564c9a"}
Oct 30 20:24:54.468232 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.468083    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jswll"
Oct 30 20:24:54.468232 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.468113    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f07c61b9e5174fa98931375533cff3886468ac1adbb1f51fdd7d9293af564c9a"
Oct 30 20:24:54.470202 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.470176    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7" event={"ID":"3c1187a4-aba6-4d72-85ac-9e464a1c688a","Type":"ContainerDied","Data":"f1d661cdef497e5abcbb4d198dc8658e3e54c2cf34e00a6bb16ef38de38106e6"}
Oct 30 20:24:54.470331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.470216    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d661cdef497e5abcbb4d198dc8658e3e54c2cf34e00a6bb16ef38de38106e6"
Oct 30 20:24:54.470331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.470278    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cncv7"
Oct 30 20:24:54.472215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.472176    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82" event={"ID":"a1ccabc5-fd7d-4358-b4d7-55124d94baa3","Type":"ContainerDied","Data":"fabc6d2f6899d150dad30f946152d6cdf175b8a9e27104a2087af28f5c8ea48e"}
Oct 30 20:24:54.472215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.472210    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bzb82"
Oct 30 20:24:54.472351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.472231    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fabc6d2f6899d150dad30f946152d6cdf175b8a9e27104a2087af28f5c8ea48e"
Oct 30 20:24:54.597598 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.597580    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:54.758063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.758043    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-bundle\") pod \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") "
Oct 30 20:24:54.758179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.758086    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-kube-api-access-tt7jm\" (UniqueName: \"kubernetes.io/projected/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-kube-api-access-tt7jm\") pod \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") "
Oct 30 20:24:54.758179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.758130    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-util\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-util\") pod \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\" (UID: \"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e\") "
Oct 30 20:24:54.758716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.758662    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-bundle" pod "9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" (UID: "9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:54.760151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.760126    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-kube-api-access-tt7jm" pod "9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" (UID: "9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e"). InnerVolumeSpecName "kube-api-access-tt7jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:54.763304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.763286    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-util" pod "9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" (UID: "9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:54.859039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.859016    5002 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-util\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:54.859039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.859036    5002 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:54.859171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:54.859045    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt7jm\" (UniqueName: \"kubernetes.io/projected/9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e-kube-api-access-tt7jm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:55.477919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:55.477894    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk" event={"ID":"9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e","Type":"ContainerDied","Data":"8c602f44bf2f4c3fe73ef5044066990eedfcc4c921ee3a111e296c32b607a845"}
Oct 30 20:24:55.478239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:55.477929    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck58nk"
Oct 30 20:24:55.478239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:55.477933    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c602f44bf2f4c3fe73ef5044066990eedfcc4c921ee3a111e296c32b607a845"
Oct 30 20:24:57.413490 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:24:57.413455    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:00.001297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001267    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"]
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001561    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001572    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001585    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerName="util"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001589    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerName="util"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001595    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerName="util"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001599    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerName="util"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001606    5002 cpu_manager.go:414] "containerMap: removing container" podUID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001610    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001622    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerName="util"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001627    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerName="util"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001632    5002 cpu_manager.go:414] "containerMap: removing container" podUID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001637    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001641    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001646    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001656    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001660    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerName="pull"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001666    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerName="extract"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001670    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerName="extract"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001675    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerName="extract"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001680    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerName="extract"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001686    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerName="extract"
Oct 30 20:25:00.001680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001690    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001696    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001700    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001708    5002 cpu_manager.go:414] "containerMap: removing container" podUID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001712    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001717    5002 cpu_manager.go:414] "containerMap: removing container" podUID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001721    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001729    5002 cpu_manager.go:414] "containerMap: removing container" podUID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001733    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001739    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001743    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001748    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerName="pull"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001752    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerName="pull"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001756    5002 cpu_manager.go:414] "containerMap: removing container" podUID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerName="pull"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001760    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerName="pull"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001766    5002 cpu_manager.go:414] "containerMap: removing container" podUID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001770    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerName="util"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001775    5002 cpu_manager.go:414] "containerMap: removing container" podUID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001779    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001785    5002 cpu_manager.go:414] "containerMap: removing container" podUID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001788    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001846    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b15d19-4f0c-4c40-8424-335e49fabb66" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001852    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="04026416-aa2c-4595-bf79-22d1ee78ea2c" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001862    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ccabc5-fd7d-4358-b4d7-55124d94baa3" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001867    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1187a4-aba6-4d72-85ac-9e464a1c688a" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001872    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6425949-c8d7-415a-83f5-eb52f5ea3463" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001877    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc2d89c-7ecd-4f41-9d8c-ebb6e47e3e8e" containerName="extract"
Oct 30 20:25:00.002254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.001882    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ac82d8-3339-4323-895e-40c552bd5e9d" containerName="extract"
Oct 30 20:25:00.003720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.003705    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"
Oct 30 20:25:00.097873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.097849    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zc2q\" (UniqueName: \"kubernetes.io/projected/6eb31bb0-64ac-495f-a6b6-353e913ee03f-kube-api-access-6zc2q\") pod \"limitador-operator-controller-manager-5778bb4449-2pc5v\" (UID: \"6eb31bb0-64ac-495f-a6b6-353e913ee03f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"
Oct 30 20:25:00.109930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.109886    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:25:00.109930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.109916    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:25:00.110077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.110033    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-c8ktn\""
Oct 30 20:25:00.111574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.111552    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"]
Oct 30 20:25:00.198795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.198772    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zc2q\" (UniqueName: \"kubernetes.io/projected/6eb31bb0-64ac-495f-a6b6-353e913ee03f-kube-api-access-6zc2q\") pod \"limitador-operator-controller-manager-5778bb4449-2pc5v\" (UID: \"6eb31bb0-64ac-495f-a6b6-353e913ee03f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"
Oct 30 20:25:00.256165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.256104    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zc2q\" (UniqueName: \"kubernetes.io/projected/6eb31bb0-64ac-495f-a6b6-353e913ee03f-kube-api-access-6zc2q\") pod \"limitador-operator-controller-manager-5778bb4449-2pc5v\" (UID: \"6eb31bb0-64ac-495f-a6b6-353e913ee03f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"
Oct 30 20:25:00.314659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.314631    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"
Oct 30 20:25:00.573140 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.573036    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:00.751575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:00.751547    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"]
Oct 30 20:25:00.753519 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:00.753496    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb31bb0_64ac_495f_a6b6_353e913ee03f.slice/crio-709af40ab28e19bd491e71bdfc09f3291f25aa92874ba56293580498936a9825 WatchSource:0}: Error finding container 709af40ab28e19bd491e71bdfc09f3291f25aa92874ba56293580498936a9825: Status 404 returned error can't find the container with id 709af40ab28e19bd491e71bdfc09f3291f25aa92874ba56293580498936a9825
Oct 30 20:25:01.198969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:01.198942    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:01.394502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:01.394475    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:01.503163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:01.503132    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v" event={"ID":"6eb31bb0-64ac-495f-a6b6-353e913ee03f","Type":"ContainerStarted","Data":"709af40ab28e19bd491e71bdfc09f3291f25aa92874ba56293580498936a9825"}
Oct 30 20:25:02.175445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:02.175390    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:04.518343 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:04.518269    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v" event={"ID":"6eb31bb0-64ac-495f-a6b6-353e913ee03f","Type":"ContainerStarted","Data":"379f57bda351eb7d0e9589d0c51dd69e8fdc1d0f6cddc60d8c2ff7596c2c3b06"}
Oct 30 20:25:04.518746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:04.518560    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"
Oct 30 20:25:04.634408 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:04.634361    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v" podStartSLOduration=2.164633986 podStartE2EDuration="5.634343565s" totalImagesPullingTime="3.469709579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:04.632874477 +0000 UTC m=+330.781925464" watchObservedRunningTime="2026-10-30 20:25:04.634343565 +0000 UTC m=+330.783394553"
Oct 30 20:25:06.578560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:06.578497    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:06.809501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:06.809441    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:07.113711 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:25:07.113680    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:25:07.509964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.509933    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"]
Oct 30 20:25:07.513295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.513276    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.621259 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.621229    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vhm96\""
Oct 30 20:25:07.622859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.622843    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"]
Oct 30 20:25:07.657163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.657142    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pprmd\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.657267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.657188    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psn8b\" (UniqueName: \"kubernetes.io/projected/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-kube-api-access-psn8b\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pprmd\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.757934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.757910    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psn8b\" (UniqueName: \"kubernetes.io/projected/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-kube-api-access-psn8b\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pprmd\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.758309 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.757974    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pprmd\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.758391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.758344    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pprmd\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.816004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.815946    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:07.816983 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.816957    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psn8b\" (UniqueName: \"kubernetes.io/projected/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-kube-api-access-psn8b\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pprmd\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.825512 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.825492    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:07.914873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.913800    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"]
Oct 30 20:25:07.918977 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.918952    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:07.972857 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.972832    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:25:07.973084 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.973060    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:25:07.976314 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.976297    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 20:25:07.979817 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:07.979803    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 20:25:08.007937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.007919    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:08.016472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.016452    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"]
Oct 30 20:25:08.059790 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.059773    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5fw\" (UniqueName: \"kubernetes.io/projected/561b5ad8-44f2-458d-b7a0-f831dd85d224-kube-api-access-fz5fw\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.059882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.059799    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.059882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.059819    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/561b5ad8-44f2-458d-b7a0-f831dd85d224-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.059882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.059846    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.059982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.059970    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.060014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.059997    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/561b5ad8-44f2-458d-b7a0-f831dd85d224-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.079231 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.079188    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:25:08.079231 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.079199    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-trbjm\""
Oct 30 20:25:08.161165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.161143    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.161255 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.161175    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/561b5ad8-44f2-458d-b7a0-f831dd85d224-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.161255 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.161207    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5fw\" (UniqueName: \"kubernetes.io/projected/561b5ad8-44f2-458d-b7a0-f831dd85d224-kube-api-access-fz5fw\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.161338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.161250    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.161338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.161282    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/561b5ad8-44f2-458d-b7a0-f831dd85d224-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.161338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.161324    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.161726 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.161693    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/561b5ad8-44f2-458d-b7a0-f831dd85d224-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.163449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.163431    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/561b5ad8-44f2-458d-b7a0-f831dd85d224-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.163768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.163751    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.163871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.163784    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.164020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.164003    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/561b5ad8-44f2-458d-b7a0-f831dd85d224-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.223788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.223767    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5fw\" (UniqueName: \"kubernetes.io/projected/561b5ad8-44f2-458d-b7a0-f831dd85d224-kube-api-access-fz5fw\") pod \"opendatahub-operator-controller-manager-585d4dbb69-4sz5n\" (UID: \"561b5ad8-44f2-458d-b7a0-f831dd85d224\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.234615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.234596    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:08.256443 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.256423    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"]
Oct 30 20:25:08.257489 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:08.257465    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b14f857_2b2e_4ea1_9b28_0e5d240cedcb.slice/crio-3f427419007a67e6883092d77aa90c681efd0a34b0536e2def228da5531ea3cb WatchSource:0}: Error finding container 3f427419007a67e6883092d77aa90c681efd0a34b0536e2def228da5531ea3cb: Status 404 returned error can't find the container with id 3f427419007a67e6883092d77aa90c681efd0a34b0536e2def228da5531ea3cb
Oct 30 20:25:08.273991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.273968    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:08.458621 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.458555    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:08.535622 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.535597    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd" event={"ID":"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb","Type":"ContainerStarted","Data":"3f427419007a67e6883092d77aa90c681efd0a34b0536e2def228da5531ea3cb"}
Oct 30 20:25:08.644422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.644385    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:08.678859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.678836    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"]
Oct 30 20:25:08.680838 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:08.680817    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561b5ad8_44f2_458d_b7a0_f831dd85d224.slice/crio-84e2e337ab598f9501e3abf69ffbc7b5bdb583b867db7fff4bf41134547799c5 WatchSource:0}: Error finding container 84e2e337ab598f9501e3abf69ffbc7b5bdb583b867db7fff4bf41134547799c5: Status 404 returned error can't find the container with id 84e2e337ab598f9501e3abf69ffbc7b5bdb583b867db7fff4bf41134547799c5
Oct 30 20:25:08.874596 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:08.874562    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:09.094629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:09.094604    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:09.541458 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:09.541428    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n" event={"ID":"561b5ad8-44f2-458d-b7a0-f831dd85d224","Type":"ContainerStarted","Data":"84e2e337ab598f9501e3abf69ffbc7b5bdb583b867db7fff4bf41134547799c5"}
Oct 30 20:25:11.283247 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:11.283206    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:12.560608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:12.560562    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n" event={"ID":"561b5ad8-44f2-458d-b7a0-f831dd85d224","Type":"ContainerStarted","Data":"321f24de4bc0c942d955c68534863f2f19ad589af41d03ff9462b771db99a677"}
Oct 30 20:25:12.820069 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:12.820031    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"]
Oct 30 20:25:12.824892 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:12.824863    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"
Oct 30 20:25:12.877863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:12.877837    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-v6qg2\""
Oct 30 20:25:12.899877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:12.899855    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcg7\" (UniqueName: \"kubernetes.io/projected/983d0d73-eccc-45f1-b2ea-3a12dd3b7860-kube-api-access-wxcg7\") pod \"authorino-operator-7b7bb4bc77-w6xjc\" (UID: \"983d0d73-eccc-45f1-b2ea-3a12dd3b7860\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"
Oct 30 20:25:13.001248 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.001213    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcg7\" (UniqueName: \"kubernetes.io/projected/983d0d73-eccc-45f1-b2ea-3a12dd3b7860-kube-api-access-wxcg7\") pod \"authorino-operator-7b7bb4bc77-w6xjc\" (UID: \"983d0d73-eccc-45f1-b2ea-3a12dd3b7860\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"
Oct 30 20:25:13.042283 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.042251    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:13.059557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.059513    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"]
Oct 30 20:25:13.066507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.066482    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcg7\" (UniqueName: \"kubernetes.io/projected/983d0d73-eccc-45f1-b2ea-3a12dd3b7860-kube-api-access-wxcg7\") pod \"authorino-operator-7b7bb4bc77-w6xjc\" (UID: \"983d0d73-eccc-45f1-b2ea-3a12dd3b7860\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"
Oct 30 20:25:13.139415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.139347    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"
Oct 30 20:25:13.519081 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.519054    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"]
Oct 30 20:25:13.520724 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:13.520702    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983d0d73_eccc_45f1_b2ea_3a12dd3b7860.slice/crio-e05b777eccbac564075d32636948364c2e8ca83f7cad51c98db41d079cf91c8f WatchSource:0}: Error finding container e05b777eccbac564075d32636948364c2e8ca83f7cad51c98db41d079cf91c8f: Status 404 returned error can't find the container with id e05b777eccbac564075d32636948364c2e8ca83f7cad51c98db41d079cf91c8f
Oct 30 20:25:13.566373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.566348    5002 generic.go:356] "Generic (PLEG): container finished" podID="561b5ad8-44f2-458d-b7a0-f831dd85d224" containerID="321f24de4bc0c942d955c68534863f2f19ad589af41d03ff9462b771db99a677" exitCode=0
Oct 30 20:25:13.566676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.566461    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n" event={"ID":"561b5ad8-44f2-458d-b7a0-f831dd85d224","Type":"ContainerDied","Data":"321f24de4bc0c942d955c68534863f2f19ad589af41d03ff9462b771db99a677"}
Oct 30 20:25:13.568310 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.568286    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd" event={"ID":"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb","Type":"ContainerStarted","Data":"aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d"}
Oct 30 20:25:13.568401 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.568384    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:13.569500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.569484    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc" event={"ID":"983d0d73-eccc-45f1-b2ea-3a12dd3b7860","Type":"ContainerStarted","Data":"e05b777eccbac564075d32636948364c2e8ca83f7cad51c98db41d079cf91c8f"}
Oct 30 20:25:13.872082 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.872037    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd" podStartSLOduration=1.721061776 podStartE2EDuration="6.872024173s" totalImagesPullingTime="5.150962397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:13.869245115 +0000 UTC m=+340.018296102" watchObservedRunningTime="2026-10-30 20:25:13.872024173 +0000 UTC m=+340.021075159"
Oct 30 20:25:13.995023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:13.994991    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:14.208509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:14.208434    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:14.577111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:14.577061    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n" event={"ID":"561b5ad8-44f2-458d-b7a0-f831dd85d224","Type":"ContainerStarted","Data":"083cbc3c61ac02456221f232abbbdf0031d0f616996bbe055ef026fb6388908a"}
Oct 30 20:25:14.577508 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:14.577159    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:14.582313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:14.582286    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:14.716382 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:14.716338    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n" podStartSLOduration=2.689354564 podStartE2EDuration="7.716324565s" totalImagesPullingTime="4.435710254s" totalInitContainerRuntime="591.259747ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:14.714382316 +0000 UTC m=+340.863433316" watchObservedRunningTime="2026-10-30 20:25:14.716324565 +0000 UTC m=+340.865375552"
Oct 30 20:25:15.525486 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:15.525457    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2pc5v"
Oct 30 20:25:15.666178 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:15.666145    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:16.592370 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:16.592338    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc" event={"ID":"983d0d73-eccc-45f1-b2ea-3a12dd3b7860","Type":"ContainerStarted","Data":"19a89087dc7899af4b960b07a120ff2a41b1678f8268953b3324ced8e5eec327"}
Oct 30 20:25:16.592597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:16.592467    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"
Oct 30 20:25:16.715352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:16.715309    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc" podStartSLOduration=2.322545352 podStartE2EDuration="4.715296932s" totalImagesPullingTime="2.39275158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:16.714734758 +0000 UTC m=+342.863785746" watchObservedRunningTime="2026-10-30 20:25:16.715296932 +0000 UTC m=+342.864347999"
Oct 30 20:25:18.578733 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.578701    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:18.774872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.774841    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:18.913050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.912983    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"]
Oct 30 20:25:18.917592 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.917574    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:18.942979 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.942949    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8ss\" (UniqueName: \"kubernetes.io/projected/57872e61-5477-4c32-a604-3db51c409fd6-kube-api-access-qj8ss\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k\" (UID: \"57872e61-5477-4c32-a604-3db51c409fd6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:18.943099 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.942993    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57872e61-5477-4c32-a604-3db51c409fd6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k\" (UID: \"57872e61-5477-4c32-a604-3db51c409fd6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:18.971062 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.971040    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 20:25:18.971646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:18.971632    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9k9xg\""
Oct 30 20:25:19.018849 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.018824    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:19.021696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.021678    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:25:19.023763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.023746    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"]
Oct 30 20:25:19.043933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.043909    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57872e61-5477-4c32-a604-3db51c409fd6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k\" (UID: \"57872e61-5477-4c32-a604-3db51c409fd6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:19.044028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.044006    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8ss\" (UniqueName: \"kubernetes.io/projected/57872e61-5477-4c32-a604-3db51c409fd6-kube-api-access-qj8ss\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k\" (UID: \"57872e61-5477-4c32-a604-3db51c409fd6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:19.044258 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.044241    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57872e61-5477-4c32-a604-3db51c409fd6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k\" (UID: \"57872e61-5477-4c32-a604-3db51c409fd6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:19.105314 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.105291    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8ss\" (UniqueName: \"kubernetes.io/projected/57872e61-5477-4c32-a604-3db51c409fd6-kube-api-access-qj8ss\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k\" (UID: \"57872e61-5477-4c32-a604-3db51c409fd6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:19.228023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.227999    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"
Oct 30 20:25:19.700090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:19.700064    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k"]
Oct 30 20:25:19.701847 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:19.701822    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57872e61_5477_4c32_a604_3db51c409fd6.slice/crio-7d4cc6160f035aee75b0673964bac64ebb545a9521f79d971a10564a74548693 WatchSource:0}: Error finding container 7d4cc6160f035aee75b0673964bac64ebb545a9521f79d971a10564a74548693: Status 404 returned error can't find the container with id 7d4cc6160f035aee75b0673964bac64ebb545a9521f79d971a10564a74548693
Oct 30 20:25:20.611862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:20.611828    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k" event={"ID":"57872e61-5477-4c32-a604-3db51c409fd6","Type":"ContainerStarted","Data":"7d4cc6160f035aee75b0673964bac64ebb545a9521f79d971a10564a74548693"}
Oct 30 20:25:21.490587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:21.490551    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:21.672042 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:21.672016    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:23.628197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:23.628167    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k" event={"ID":"57872e61-5477-4c32-a604-3db51c409fd6","Type":"ContainerStarted","Data":"d6221b411ceda2418bd7c6b35a9355d966ff3a6cd5015aed91d07b6b5c7668a7"}
Oct 30 20:25:23.851453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:23.851407    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hkt5k" podStartSLOduration=2.877424381 podStartE2EDuration="5.851394726s" totalImagesPullingTime="2.973970345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:23.800184799 +0000 UTC m=+349.949235799" watchObservedRunningTime="2026-10-30 20:25:23.851394726 +0000 UTC m=+350.000445712"
Oct 30 20:25:24.541325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:24.541288    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:24.579937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:24.579909    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:24.762642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:24.762613    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:25.587234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:25.587208    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-4sz5n"
Oct 30 20:25:26.026319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.026288    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"]
Oct 30 20:25:26.030897 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.030881    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.050830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.050807    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:26.093153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.093128    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"]
Oct 30 20:25:26.093423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.093401    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d66661c-3d8b-4af2-af19-00af4f63c714-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-br5mh\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.093474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.093433    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8js5m\" (UniqueName: \"kubernetes.io/projected/4d66661c-3d8b-4af2-af19-00af4f63c714-kube-api-access-8js5m\") pod \"kuadrant-operator-controller-manager-678d87766f-br5mh\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.194291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.194267    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d66661c-3d8b-4af2-af19-00af4f63c714-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-br5mh\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.194409 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.194297    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8js5m\" (UniqueName: \"kubernetes.io/projected/4d66661c-3d8b-4af2-af19-00af4f63c714-kube-api-access-8js5m\") pod \"kuadrant-operator-controller-manager-678d87766f-br5mh\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.194674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.194657    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d66661c-3d8b-4af2-af19-00af4f63c714-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-br5mh\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.257643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.257624    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8js5m\" (UniqueName: \"kubernetes.io/projected/4d66661c-3d8b-4af2-af19-00af4f63c714-kube-api-access-8js5m\") pod \"kuadrant-operator-controller-manager-678d87766f-br5mh\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.342940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.342887    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:26.762825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.762802    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"]
Oct 30 20:25:26.764843 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:26.764815    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d66661c_3d8b_4af2_af19_00af4f63c714.slice/crio-1fce9fbc4dce795b0b748a9e157cfa786d9a63fe806bb1d620ce78ce8fc4c975 WatchSource:0}: Error finding container 1fce9fbc4dce795b0b748a9e157cfa786d9a63fe806bb1d620ce78ce8fc4c975: Status 404 returned error can't find the container with id 1fce9fbc4dce795b0b748a9e157cfa786d9a63fe806bb1d620ce78ce8fc4c975
Oct 30 20:25:26.809726 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:26.809703    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:27.004808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.004733    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:27.389921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.389847    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:27.599613 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.599585    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w6xjc"
Oct 30 20:25:27.624496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.624463    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:27.653022 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.652963    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh" event={"ID":"4d66661c-3d8b-4af2-af19-00af4f63c714","Type":"ContainerStarted","Data":"12d192caec54458bb3b0356c9da40f32e12dcf0e3b7c829e814352d304d1928e"}
Oct 30 20:25:27.653131 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.653038    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh" event={"ID":"4d66661c-3d8b-4af2-af19-00af4f63c714","Type":"ContainerStarted","Data":"1fce9fbc4dce795b0b748a9e157cfa786d9a63fe806bb1d620ce78ce8fc4c975"}
Oct 30 20:25:27.653131 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.653060    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:27.858506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:27.858456    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh" podStartSLOduration=2.858444581 podStartE2EDuration="2.858444581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:27.857087177 +0000 UTC m=+354.006138163" watchObservedRunningTime="2026-10-30 20:25:27.858444581 +0000 UTC m=+354.007495568"
Oct 30 20:25:28.014779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:28.014753    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:28.220622 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:28.220591    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:30.293290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:30.293264    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:30.478207 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:30.478186    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:32.548998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:32.548969    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:32.732187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:32.732159    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:34.278557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:34.278500    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:36.480464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:36.480401    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:36.677544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:36.677501    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:37.063377 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:37.063343    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:37.113352 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:25:37.113324    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:25:37.982046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:37.981986    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:38.156804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.156772    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:38.384443 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.384361    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:38.561892 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.561862    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:38.659622 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.659552    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 20:25:38.754862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.754832    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:38.891052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.891024    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"]
Oct 30 20:25:38.891265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.891244    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd" podUID="0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" containerName="manager" containerID="cri-o://aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d" gracePeriod=10
Oct 30 20:25:38.982395 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:38.982366    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:39.127114 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.127066    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"]
Oct 30 20:25:39.135425 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.135403    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.187775 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.187749    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.187898 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.187777    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.187898 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.187816    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.187988 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.187955    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.188030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.187989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.188030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.188014    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.188142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.188125    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.188188 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.188151    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38948f25-fabf-4bf8-8e26-4520b69f1844-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.188188 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.188167    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfdn\" (UniqueName: \"kubernetes.io/projected/38948f25-fabf-4bf8-8e26-4520b69f1844-kube-api-access-pxfdn\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.189451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.189435    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-h87sm\""
Oct 30 20:25:39.196639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.196610    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"]
Oct 30 20:25:39.196984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.196959    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:39.289044 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.288986    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289044 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289017    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289042    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289076    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289099    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38948f25-fabf-4bf8-8e26-4520b69f1844-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289223    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfdn\" (UniqueName: \"kubernetes.io/projected/38948f25-fabf-4bf8-8e26-4520b69f1844-kube-api-access-pxfdn\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289383 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289331    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289383 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289361    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289483 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289403    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289570 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289482    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289570 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289561    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289698 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289678    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289747 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289730    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.289936 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.289914    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38948f25-fabf-4bf8-8e26-4520b69f1844-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.291631 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.291614    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.291740 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.291725    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.361087 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.361059    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfdn\" (UniqueName: \"kubernetes.io/projected/38948f25-fabf-4bf8-8e26-4520b69f1844-kube-api-access-pxfdn\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.369052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.369026    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38948f25-fabf-4bf8-8e26-4520b69f1844-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp\" (UID: \"38948f25-fabf-4bf8-8e26-4520b69f1844\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.448904 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.448877    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:39.495831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.495789    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:39.748420 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.748388    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:39.893093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.893069    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"]
Oct 30 20:25:39.894268 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:39.894240    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38948f25_fabf_4bf8_8e26_4520b69f1844.slice/crio-3ad69e838e56f7578a0ddd3dac78d89a47547d7fb0fbc8a9e1bdf366af2c8875 WatchSource:0}: Error finding container 3ad69e838e56f7578a0ddd3dac78d89a47547d7fb0fbc8a9e1bdf366af2c8875: Status 404 returned error can't find the container with id 3ad69e838e56f7578a0ddd3dac78d89a47547d7fb0fbc8a9e1bdf366af2c8875
Oct 30 20:25:39.896174 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.896145    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:25:39.896243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.896205    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:25:39.896243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:39.896233    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:25:40.047854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.047776    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:40.255459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.255429    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:40.530078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.530056    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:40.599459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.599431    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-kube-api-access-psn8b\" (UniqueName: \"kubernetes.io/projected/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-kube-api-access-psn8b\") pod \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") "
Oct 30 20:25:40.599590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.599495    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-extensions-socket-volume\") pod \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\" (UID: \"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb\") "
Oct 30 20:25:40.599820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.599793    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-extensions-socket-volume" pod "0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" (UID: "0b14f857-2b2e-4ea1-9b28-0e5d240cedcb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:40.601633 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.601610    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-kube-api-access-psn8b" pod "0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" (UID: "0b14f857-2b2e-4ea1-9b28-0e5d240cedcb"). InnerVolumeSpecName "kube-api-access-psn8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:40.700426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.700352    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psn8b\" (UniqueName: \"kubernetes.io/projected/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-kube-api-access-psn8b\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:40.700426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.700371    5002 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb-extensions-socket-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:40.717359 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.717339    5002 generic.go:356] "Generic (PLEG): container finished" podID="0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" containerID="aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d" exitCode=0
Oct 30 20:25:40.717444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.717367    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd" event={"ID":"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb","Type":"ContainerDied","Data":"aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d"}
Oct 30 20:25:40.717444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.717382    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"
Oct 30 20:25:40.717444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.717410    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd" event={"ID":"0b14f857-2b2e-4ea1-9b28-0e5d240cedcb","Type":"ContainerDied","Data":"3f427419007a67e6883092d77aa90c681efd0a34b0536e2def228da5531ea3cb"}
Oct 30 20:25:40.717444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.717436    5002 scope.go:122] "RemoveContainer" containerID="aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d"
Oct 30 20:25:40.719068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.719035    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp" event={"ID":"38948f25-fabf-4bf8-8e26-4520b69f1844","Type":"ContainerStarted","Data":"e2acb92e6bc38f43b47268a292d5260981ffdcf8e25eb23a707b1f222a5fb92d"}
Oct 30 20:25:40.719152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.719105    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp" event={"ID":"38948f25-fabf-4bf8-8e26-4520b69f1844","Type":"ContainerStarted","Data":"3ad69e838e56f7578a0ddd3dac78d89a47547d7fb0fbc8a9e1bdf366af2c8875"}
Oct 30 20:25:40.727037 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.727019    5002 scope.go:122] "RemoveContainer" containerID="aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d"
Oct 30 20:25:40.727305 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:25:40.727284    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d\": container with ID starting with aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d not found: ID does not exist" containerID="aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d"
Oct 30 20:25:40.727373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.727317    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d"} err="failed to get container status \"aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d\": rpc error: code = NotFound desc = could not find container \"aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d\": container with ID starting with aa1d12b739faea7a6b1c5a68edc6c6293b9f2b55dc5a98028cd513c8a1b8559d not found: ID does not exist"
Oct 30 20:25:40.772138 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.772118    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:40.882043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.882001    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp" podStartSLOduration=1.881989499 podStartE2EDuration="1.881989499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:40.879985148 +0000 UTC m=+367.029036149" watchObservedRunningTime="2026-10-30 20:25:40.881989499 +0000 UTC m=+367.031040487"
Oct 30 20:25:40.998259 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:40.998198    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:41.000595 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:41.000573    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"]
Oct 30 20:25:41.005925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:41.005905    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pprmd"]
Oct 30 20:25:41.449432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:41.449378    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:41.453764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:41.453744    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:41.724806 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:41.724778    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:41.725669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:41.725650    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp"
Oct 30 20:25:41.964764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:41.964738    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" path="/var/lib/kubelet/pods/0b14f857-2b2e-4ea1-9b28-0e5d240cedcb/volumes"
Oct 30 20:25:43.394484 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:43.394443    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:43.583708 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:43.583674    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:46.428882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:46.428838    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:46.742611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:46.742586    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:46.929282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:46.929249    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:51.459510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.459482    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"]
Oct 30 20:25:51.459961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.459824    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" containerName="manager"
Oct 30 20:25:51.459961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.459835    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" containerName="manager"
Oct 30 20:25:51.459961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.459913    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b14f857-2b2e-4ea1-9b28-0e5d240cedcb" containerName="manager"
Oct 30 20:25:51.464889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.464871    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.518180 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.518160    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 20:25:51.518180 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.518173    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 20:25:51.587024 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.587002    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/887c992d-ed9b-45a6-9d96-4cec589782db-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.587126 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.587035    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5l9d\" (UniqueName: \"kubernetes.io/projected/887c992d-ed9b-45a6-9d96-4cec589782db-kube-api-access-q5l9d\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.587126 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.587067    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/887c992d-ed9b-45a6-9d96-4cec589782db-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.587200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.587183    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887c992d-ed9b-45a6-9d96-4cec589782db-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.669998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.669969    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gm7dt\""
Oct 30 20:25:51.669998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.669986    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 20:25:51.671953 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.671934    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"]
Oct 30 20:25:51.688325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.688304    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/887c992d-ed9b-45a6-9d96-4cec589782db-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.688418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.688331    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5l9d\" (UniqueName: \"kubernetes.io/projected/887c992d-ed9b-45a6-9d96-4cec589782db-kube-api-access-q5l9d\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.688418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.688352    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/887c992d-ed9b-45a6-9d96-4cec589782db-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.688418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.688388    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887c992d-ed9b-45a6-9d96-4cec589782db-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.688739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.688722    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/887c992d-ed9b-45a6-9d96-4cec589782db-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.688800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.688767    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/887c992d-ed9b-45a6-9d96-4cec589782db-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.689149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.689132    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/887c992d-ed9b-45a6-9d96-4cec589782db-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.746024 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.745974    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5l9d\" (UniqueName: \"kubernetes.io/projected/887c992d-ed9b-45a6-9d96-4cec589782db-kube-api-access-q5l9d\") pod \"kserve-module-controller-manager-7566cf5fc8-g97s6\" (UID: \"887c992d-ed9b-45a6-9d96-4cec589782db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:51.775160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:51.775143    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:52.193221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:52.193199    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"]
Oct 30 20:25:52.194718 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:25:52.194692    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887c992d_ed9b_45a6_9d96_4cec589782db.slice/crio-a5babfef2571d70f4bb7a530666a943e6316166563eb581d945590dd3ddaa473 WatchSource:0}: Error finding container a5babfef2571d70f4bb7a530666a943e6316166563eb581d945590dd3ddaa473: Status 404 returned error can't find the container with id a5babfef2571d70f4bb7a530666a943e6316166563eb581d945590dd3ddaa473
Oct 30 20:25:52.776656 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:52.776625    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6" event={"ID":"887c992d-ed9b-45a6-9d96-4cec589782db","Type":"ContainerStarted","Data":"a5babfef2571d70f4bb7a530666a943e6316166563eb581d945590dd3ddaa473"}
Oct 30 20:25:55.591786 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:55.591755    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:55.784955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:55.784922    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:55.794359 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:55.794328    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6" event={"ID":"887c992d-ed9b-45a6-9d96-4cec589782db","Type":"ContainerStarted","Data":"772a80436e2757206755603a27d38ffc5895ca42df30b3cfb6870dbaee7758a4"}
Oct 30 20:25:55.794502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:55.794380    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:25:55.975969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:55.975926    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6" podStartSLOduration=2.292664879 podStartE2EDuration="4.975914412s" totalImagesPullingTime="2.683249533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:55.975356697 +0000 UTC m=+382.124407687" watchObservedRunningTime="2026-10-30 20:25:55.975914412 +0000 UTC m=+382.124965398"
Oct 30 20:25:57.393643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:57.393612    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:57.574930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:57.574899    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:57.746725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:57.746697    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:58.055755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:58.055680    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:25:58.270870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:25:58.270841    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:00.824057 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.824027    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qt4k"]
Oct 30 20:26:00.826362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.826346    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:26:00.826471 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.826453    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k"
Oct 30 20:26:00.828315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.828293    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:00.883676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.883653    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2p4vw\""
Oct 30 20:26:00.884915 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.884889    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qt4k"]
Oct 30 20:26:00.933876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.933855    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 20:26:00.935672 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.935654    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:26:00.963388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.963367    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjpj\" (UniqueName: \"kubernetes.io/projected/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc-kube-api-access-6zjpj\") pod \"authorino-56c6c9ddd5-4qt4k\" (UID: \"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k"
Oct 30 20:26:00.963474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.963412    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f5998c48-f172-4a4a-ad2c-2687a3e216b9-config-file\") pod \"limitador-limitador-7c9b87d574-bh9rr\" (UID: \"f5998c48-f172-4a4a-ad2c-2687a3e216b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:00.963510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.963479    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2rk\" (UniqueName: \"kubernetes.io/projected/f5998c48-f172-4a4a-ad2c-2687a3e216b9-kube-api-access-6v2rk\") pod \"limitador-limitador-7c9b87d574-bh9rr\" (UID: \"f5998c48-f172-4a4a-ad2c-2687a3e216b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:00.984414 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:00.984398    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rjp9d\""
Oct 30 20:26:01.063766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.063745    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2rk\" (UniqueName: \"kubernetes.io/projected/f5998c48-f172-4a4a-ad2c-2687a3e216b9-kube-api-access-6v2rk\") pod \"limitador-limitador-7c9b87d574-bh9rr\" (UID: \"f5998c48-f172-4a4a-ad2c-2687a3e216b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:01.063855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.063792    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjpj\" (UniqueName: \"kubernetes.io/projected/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc-kube-api-access-6zjpj\") pod \"authorino-56c6c9ddd5-4qt4k\" (UID: \"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k"
Oct 30 20:26:01.063855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.063826    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f5998c48-f172-4a4a-ad2c-2687a3e216b9-config-file\") pod \"limitador-limitador-7c9b87d574-bh9rr\" (UID: \"f5998c48-f172-4a4a-ad2c-2687a3e216b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:01.064326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.064310    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f5998c48-f172-4a4a-ad2c-2687a3e216b9-config-file\") pod \"limitador-limitador-7c9b87d574-bh9rr\" (UID: \"f5998c48-f172-4a4a-ad2c-2687a3e216b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:01.122801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.122746    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2rk\" (UniqueName: \"kubernetes.io/projected/f5998c48-f172-4a4a-ad2c-2687a3e216b9-kube-api-access-6v2rk\") pod \"limitador-limitador-7c9b87d574-bh9rr\" (UID: \"f5998c48-f172-4a4a-ad2c-2687a3e216b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:01.123210 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.123194    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjpj\" (UniqueName: \"kubernetes.io/projected/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc-kube-api-access-6zjpj\") pod \"authorino-56c6c9ddd5-4qt4k\" (UID: \"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k"
Oct 30 20:26:01.145615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.145596    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k"
Oct 30 20:26:01.147705 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.147425    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:26:01.151656 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.151639    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:01.518221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.518193    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qt4k"]
Oct 30 20:26:01.555592 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:26:01.555568    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c8f9868_aeb7_4d42_b0e9_ae8e96572cdc.slice/crio-d89e24fedd988153ed37466d6080d2537d16ee0bf3a359817b59bdbed2921737 WatchSource:0}: Error finding container d89e24fedd988153ed37466d6080d2537d16ee0bf3a359817b59bdbed2921737: Status 404 returned error can't find the container with id d89e24fedd988153ed37466d6080d2537d16ee0bf3a359817b59bdbed2921737
Oct 30 20:26:01.567603 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.567583    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:26:01.605301 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:26:01.605275    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5998c48_f172_4a4a_ad2c_2687a3e216b9.slice/crio-7c7460e02811752b3c29bf89fe5f61602824e3e5f477c5bf9fe5ef0be7743628 WatchSource:0}: Error finding container 7c7460e02811752b3c29bf89fe5f61602824e3e5f477c5bf9fe5ef0be7743628: Status 404 returned error can't find the container with id 7c7460e02811752b3c29bf89fe5f61602824e3e5f477c5bf9fe5ef0be7743628
Oct 30 20:26:01.825959 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.825900    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr" event={"ID":"f5998c48-f172-4a4a-ad2c-2687a3e216b9","Type":"ContainerStarted","Data":"7c7460e02811752b3c29bf89fe5f61602824e3e5f477c5bf9fe5ef0be7743628"}
Oct 30 20:26:01.827103 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:01.827084    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k" event={"ID":"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc","Type":"ContainerStarted","Data":"d89e24fedd988153ed37466d6080d2537d16ee0bf3a359817b59bdbed2921737"}
Oct 30 20:26:02.395786 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:02.395753    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:02.584304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:02.584254    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:03.839162 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:03.839136    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr" event={"ID":"f5998c48-f172-4a4a-ad2c-2687a3e216b9","Type":"ContainerStarted","Data":"93caef6beb9c484982beec9a02364f72833efbcdc0c9ed35279bd64696dd059f"}
Oct 30 20:26:03.839442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:03.839311    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:03.958278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:03.958228    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr" podStartSLOduration=2.3054536629999998 podStartE2EDuration="3.95821463s" totalImagesPullingTime="1.652760967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:03.956586806 +0000 UTC m=+390.105637793" watchObservedRunningTime="2026-10-30 20:26:03.95821463 +0000 UTC m=+390.107265664"
Oct 30 20:26:04.247194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:04.247162    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:04.445685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:04.445655    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:04.845243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:04.845210    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k" event={"ID":"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc","Type":"ContainerStarted","Data":"12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6"}
Oct 30 20:26:04.967242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:04.967197    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k" podStartSLOduration=2.727854284 podStartE2EDuration="4.967182816s" totalImagesPullingTime="2.239328532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:04.966567571 +0000 UTC m=+391.115618558" watchObservedRunningTime="2026-10-30 20:26:04.967182816 +0000 UTC m=+391.116233802"
Oct 30 20:26:05.092798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:05.092769    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:06.401282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:06.401251    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:06.584338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:06.584304    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:06.797037 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:06.797007    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:06.982076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:06.982042    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:07.114084 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:26:07.113998    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:26:07.162549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:07.162508    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:14.846740 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:14.846666    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"
Oct 30 20:26:15.808257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:15.808225    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:16.007094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:16.007067    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:16.800029 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:16.800008    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g97s6"
Oct 30 20:26:18.238579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:18.238527    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:18.417212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:18.417184    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:25.931182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:25.931133    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:26.134868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:26.134835    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:26.331757 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:26.331728    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:26.519266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:26.519234    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:32.290213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:32.290181    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:32.498043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:32.498010    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:35.359352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:35.359318    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:35.541038 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:35.540997    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:37.114777 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:26:37.114742    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:26:55.889099 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:55.889066    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:56.092518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:56.092484    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:56.303834 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:56.303803    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:56.539733 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:56.539700    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:56.729871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:56.729840    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:56.998191 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:56.998082    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:58.162848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:26:58.162811    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:03.316650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:03.316617    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:03.512564 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:03.512519    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:03.696413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:03.696346    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:03.887475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:03.887453    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:04.138880 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:04.138853    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:04.335298 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:04.335273    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:04.528150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:04.528116    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:06.034394 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:06.034358    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:06.245728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:06.245699    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:06.438185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:06.438118    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:06.620927 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:06.620892    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:07.113899 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:27:07.113863    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:27:13.301749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:13.301715    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:27:16.642955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:16.642922    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:16.865020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:16.864984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:16.914400 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:16.914334    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r9xlz"]
Oct 30 20:27:16.917724 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:16.917709    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz"
Oct 30 20:27:16.975313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:16.975288    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r9xlz"]
Oct 30 20:27:17.052567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.052543    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wgq\" (UniqueName: \"kubernetes.io/projected/6c624afa-0fdb-4725-b66e-419aa3cb0eab-kube-api-access-76wgq\") pod \"authorino-6f6944bcdd-r9xlz\" (UID: \"6c624afa-0fdb-4725-b66e-419aa3cb0eab\") " pod="kuadrant-system/authorino-6f6944bcdd-r9xlz"
Oct 30 20:27:17.153830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.153801    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wgq\" (UniqueName: \"kubernetes.io/projected/6c624afa-0fdb-4725-b66e-419aa3cb0eab-kube-api-access-76wgq\") pod \"authorino-6f6944bcdd-r9xlz\" (UID: \"6c624afa-0fdb-4725-b66e-419aa3cb0eab\") " pod="kuadrant-system/authorino-6f6944bcdd-r9xlz"
Oct 30 20:27:17.222558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.222513    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wgq\" (UniqueName: \"kubernetes.io/projected/6c624afa-0fdb-4725-b66e-419aa3cb0eab-kube-api-access-76wgq\") pod \"authorino-6f6944bcdd-r9xlz\" (UID: \"6c624afa-0fdb-4725-b66e-419aa3cb0eab\") " pod="kuadrant-system/authorino-6f6944bcdd-r9xlz"
Oct 30 20:27:17.229300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.229270    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz"
Oct 30 20:27:17.381824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.381794    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r9xlz"]
Oct 30 20:27:17.456427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.456395    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6846644bdd-rf4kc"]
Oct 30 20:27:17.459977 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.459958    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6846644bdd-rf4kc"
Oct 30 20:27:17.525688 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.525622    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6846644bdd-rf4kc"]
Oct 30 20:27:17.557460 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.557431    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82r7t\" (UniqueName: \"kubernetes.io/projected/c22fd554-ec00-49cd-893b-6c18ac83aaae-kube-api-access-82r7t\") pod \"authorino-6846644bdd-rf4kc\" (UID: \"c22fd554-ec00-49cd-893b-6c18ac83aaae\") " pod="kuadrant-system/authorino-6846644bdd-rf4kc"
Oct 30 20:27:17.621324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.621293    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r9xlz"]
Oct 30 20:27:17.643705 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:27:17.643680    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c624afa_0fdb_4725_b66e_419aa3cb0eab.slice/crio-1270cfc0f510b1235facca4355021177c47371acae15bf0ca923a23c37135f20 WatchSource:0}: Error finding container 1270cfc0f510b1235facca4355021177c47371acae15bf0ca923a23c37135f20: Status 404 returned error can't find the container with id 1270cfc0f510b1235facca4355021177c47371acae15bf0ca923a23c37135f20
Oct 30 20:27:17.657993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.657972    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82r7t\" (UniqueName: \"kubernetes.io/projected/c22fd554-ec00-49cd-893b-6c18ac83aaae-kube-api-access-82r7t\") pod \"authorino-6846644bdd-rf4kc\" (UID: \"c22fd554-ec00-49cd-893b-6c18ac83aaae\") " pod="kuadrant-system/authorino-6846644bdd-rf4kc"
Oct 30 20:27:17.717860 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.717834    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82r7t\" (UniqueName: \"kubernetes.io/projected/c22fd554-ec00-49cd-893b-6c18ac83aaae-kube-api-access-82r7t\") pod \"authorino-6846644bdd-rf4kc\" (UID: \"c22fd554-ec00-49cd-893b-6c18ac83aaae\") " pod="kuadrant-system/authorino-6846644bdd-rf4kc"
Oct 30 20:27:17.771632 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.771611    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6846644bdd-rf4kc"
Oct 30 20:27:17.801448 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:17.801421    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6846644bdd-rf4kc"]
Oct 30 20:27:18.020165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.020135    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:54708->10.132.0.33:15021: read: connection reset by peer" start-of-body=
Oct 30 20:27:18.020378 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.020197    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:54708->10.132.0.33:15021: read: connection reset by peer"
Oct 30 20:27:18.155253 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.155226    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6846644bdd-rf4kc"]
Oct 30 20:27:18.156006 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:27:18.155980    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc22fd554_ec00_49cd_893b_6c18ac83aaae.slice/crio-408e2c331606fc901f7c7a1ce17c05f07d277185d95e510d3d58b17c78752cd0 WatchSource:0}: Error finding container 408e2c331606fc901f7c7a1ce17c05f07d277185d95e510d3d58b17c78752cd0: Status 404 returned error can't find the container with id 408e2c331606fc901f7c7a1ce17c05f07d277185d95e510d3d58b17c78752cd0
Oct 30 20:27:18.188846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.188796    5002 generic.go:356] "Generic (PLEG): container finished" podID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerID="7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee" exitCode=0
Oct 30 20:27:18.188943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.188874    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" event={"ID":"f1d89af5-52dd-4707-ab01-dd7b28381b9a","Type":"ContainerDied","Data":"7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee"}
Oct 30 20:27:18.189242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.189222    5002 scope.go:122] "RemoveContainer" containerID="7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee"
Oct 30 20:27:18.190150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.190132    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6846644bdd-rf4kc" event={"ID":"c22fd554-ec00-49cd-893b-6c18ac83aaae","Type":"ContainerStarted","Data":"408e2c331606fc901f7c7a1ce17c05f07d277185d95e510d3d58b17c78752cd0"}
Oct 30 20:27:18.190234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.190158    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:27:18.190234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.190223    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:27:18.190327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.190287    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:27:18.191305 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:18.191284    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz" event={"ID":"6c624afa-0fdb-4725-b66e-419aa3cb0eab","Type":"ContainerStarted","Data":"1270cfc0f510b1235facca4355021177c47371acae15bf0ca923a23c37135f20"}
Oct 30 20:27:19.197151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.197123    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz" event={"ID":"6c624afa-0fdb-4725-b66e-419aa3cb0eab","Type":"ContainerStarted","Data":"28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a"}
Oct 30 20:27:19.197474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.197242    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz" podUID="6c624afa-0fdb-4725-b66e-419aa3cb0eab" containerName="authorino" containerID="cri-o://28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a" gracePeriod=30
Oct 30 20:27:19.199682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.199641    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" event={"ID":"f1d89af5-52dd-4707-ab01-dd7b28381b9a","Type":"ContainerStarted","Data":"2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f"}
Oct 30 20:27:19.447518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.447468    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz" podStartSLOduration=2.010320069 podStartE2EDuration="3.447453911s" totalImagesPullingTime="1.437133842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:19.324055816 +0000 UTC m=+465.473106806" watchObservedRunningTime="2026-10-30 20:27:19.447453911 +0000 UTC m=+465.596504898"
Oct 30 20:27:19.600399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.600377    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz"
Oct 30 20:27:19.603470 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.603455    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:27:19.603647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.603629    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:27:19.603685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.603668    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 20:27:19.673021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.672957    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c624afa-0fdb-4725-b66e-419aa3cb0eab-kube-api-access-76wgq\" (UniqueName: \"kubernetes.io/projected/6c624afa-0fdb-4725-b66e-419aa3cb0eab-kube-api-access-76wgq\") pod \"6c624afa-0fdb-4725-b66e-419aa3cb0eab\" (UID: \"6c624afa-0fdb-4725-b66e-419aa3cb0eab\") "
Oct 30 20:27:19.675016 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.674995    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c624afa-0fdb-4725-b66e-419aa3cb0eab-kube-api-access-76wgq" pod "6c624afa-0fdb-4725-b66e-419aa3cb0eab" (UID: "6c624afa-0fdb-4725-b66e-419aa3cb0eab"). InnerVolumeSpecName "kube-api-access-76wgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:19.774110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:19.774080    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wgq\" (UniqueName: \"kubernetes.io/projected/6c624afa-0fdb-4725-b66e-419aa3cb0eab-kube-api-access-76wgq\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:20.206975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.206937    5002 generic.go:356] "Generic (PLEG): container finished" podID="6c624afa-0fdb-4725-b66e-419aa3cb0eab" containerID="28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a" exitCode=0
Oct 30 20:27:20.207459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.206992    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz"
Oct 30 20:27:20.207459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.207004    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz" event={"ID":"6c624afa-0fdb-4725-b66e-419aa3cb0eab","Type":"ContainerDied","Data":"28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a"}
Oct 30 20:27:20.207459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.207045    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-r9xlz" event={"ID":"6c624afa-0fdb-4725-b66e-419aa3cb0eab","Type":"ContainerDied","Data":"1270cfc0f510b1235facca4355021177c47371acae15bf0ca923a23c37135f20"}
Oct 30 20:27:20.207459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.207061    5002 scope.go:122] "RemoveContainer" containerID="28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a"
Oct 30 20:27:20.230952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.230866    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6846644bdd-rf4kc" event={"ID":"c22fd554-ec00-49cd-893b-6c18ac83aaae","Type":"ContainerStarted","Data":"41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564"}
Oct 30 20:27:20.231279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.231239    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6846644bdd-rf4kc" podUID="c22fd554-ec00-49cd-893b-6c18ac83aaae" containerName="authorino" containerID="cri-o://41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564" gracePeriod=30
Oct 30 20:27:20.245209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.244778    5002 scope.go:122] "RemoveContainer" containerID="28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a"
Oct 30 20:27:20.245209 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:27:20.245152    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a\": container with ID starting with 28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a not found: ID does not exist" containerID="28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a"
Oct 30 20:27:20.245209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.245190    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a"} err="failed to get container status \"28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a\": rpc error: code = NotFound desc = could not find container \"28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a\": container with ID starting with 28253dedd2d221dd203e66092f376b87ddc56960b866a0b3b1a877149886369a not found: ID does not exist"
Oct 30 20:27:20.392216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.390562    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6846644bdd-rf4kc" podStartSLOduration=2.099489547 podStartE2EDuration="3.390521483s" totalImagesPullingTime="1.291031936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:20.389091742 +0000 UTC m=+466.538142727" watchObservedRunningTime="2026-10-30 20:27:20.390521483 +0000 UTC m=+466.539572503"
Oct 30 20:27:20.490225 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.490171    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6846644bdd-rf4kc"
Oct 30 20:27:20.533557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.509661    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r9xlz"]
Oct 30 20:27:20.533557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.522040    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r9xlz"]
Oct 30 20:27:20.545287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.541693    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qt4k"]
Oct 30 20:27:20.545287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.541999    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k" podUID="4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc" containerName="authorino" containerID="cri-o://12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6" gracePeriod=30
Oct 30 20:27:20.580251 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.580225    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22fd554-ec00-49cd-893b-6c18ac83aaae-kube-api-access-82r7t\" (UniqueName: \"kubernetes.io/projected/c22fd554-ec00-49cd-893b-6c18ac83aaae-kube-api-access-82r7t\") pod \"c22fd554-ec00-49cd-893b-6c18ac83aaae\" (UID: \"c22fd554-ec00-49cd-893b-6c18ac83aaae\") "
Oct 30 20:27:20.582742 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.582717    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22fd554-ec00-49cd-893b-6c18ac83aaae-kube-api-access-82r7t" pod "c22fd554-ec00-49cd-893b-6c18ac83aaae" (UID: "c22fd554-ec00-49cd-893b-6c18ac83aaae"). InnerVolumeSpecName "kube-api-access-82r7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:20.604093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.604068    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:27:20.604181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.604111    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 20:27:20.681562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.681515    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82r7t\" (UniqueName: \"kubernetes.io/projected/c22fd554-ec00-49cd-893b-6c18ac83aaae-kube-api-access-82r7t\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:20.836552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.836140    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k"
Oct 30 20:27:20.883500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.883471    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc-kube-api-access-6zjpj\" (UniqueName: \"kubernetes.io/projected/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc-kube-api-access-6zjpj\") pod \"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc\" (UID: \"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc\") "
Oct 30 20:27:20.885550 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.885514    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc-kube-api-access-6zjpj" pod "4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc" (UID: "4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc"). InnerVolumeSpecName "kube-api-access-6zjpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:20.984027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:20.983995    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zjpj\" (UniqueName: \"kubernetes.io/projected/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc-kube-api-access-6zjpj\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:21.241584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.241548    5002 generic.go:356] "Generic (PLEG): container finished" podID="4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc" containerID="12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6" exitCode=0
Oct 30 20:27:21.241584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.241579    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k"
Oct 30 20:27:21.242129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.241661    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k" event={"ID":"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc","Type":"ContainerDied","Data":"12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6"}
Oct 30 20:27:21.242129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.241698    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qt4k" event={"ID":"4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc","Type":"ContainerDied","Data":"d89e24fedd988153ed37466d6080d2537d16ee0bf3a359817b59bdbed2921737"}
Oct 30 20:27:21.242129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.241720    5002 scope.go:122] "RemoveContainer" containerID="12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6"
Oct 30 20:27:21.244147 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.244126    5002 generic.go:356] "Generic (PLEG): container finished" podID="c22fd554-ec00-49cd-893b-6c18ac83aaae" containerID="41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564" exitCode=0
Oct 30 20:27:21.244263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.244166    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6846644bdd-rf4kc"
Oct 30 20:27:21.244263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.244181    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6846644bdd-rf4kc" event={"ID":"c22fd554-ec00-49cd-893b-6c18ac83aaae","Type":"ContainerDied","Data":"41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564"}
Oct 30 20:27:21.244263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.244202    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6846644bdd-rf4kc" event={"ID":"c22fd554-ec00-49cd-893b-6c18ac83aaae","Type":"ContainerDied","Data":"408e2c331606fc901f7c7a1ce17c05f07d277185d95e510d3d58b17c78752cd0"}
Oct 30 20:27:21.251184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.251167    5002 scope.go:122] "RemoveContainer" containerID="12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6"
Oct 30 20:27:21.251457 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:27:21.251436    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6\": container with ID starting with 12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6 not found: ID does not exist" containerID="12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6"
Oct 30 20:27:21.251575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.251462    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6"} err="failed to get container status \"12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6\": rpc error: code = NotFound desc = could not find container \"12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6\": container with ID starting with 12e6ec76ff7eb829437d99913ddcd14cacd91bbeed608a860951d57f2a942dd6 not found: ID does not exist"
Oct 30 20:27:21.251575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.251481    5002 scope.go:122] "RemoveContainer" containerID="41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564"
Oct 30 20:27:21.259749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.259733    5002 scope.go:122] "RemoveContainer" containerID="41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564"
Oct 30 20:27:21.259989 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:27:21.259974    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564\": container with ID starting with 41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564 not found: ID does not exist" containerID="41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564"
Oct 30 20:27:21.260043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.259994    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564"} err="failed to get container status \"41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564\": rpc error: code = NotFound desc = could not find container \"41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564\": container with ID starting with 41dc01aab80e05d587919855f819197ab6523582799a65837f32cbf8d7ae9564 not found: ID does not exist"
Oct 30 20:27:21.409408 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.409385    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qt4k"]
Oct 30 20:27:21.432554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.421860    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qt4k"]
Oct 30 20:27:21.536654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.536566    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6846644bdd-rf4kc"]
Oct 30 20:27:21.539194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.539167    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6846644bdd-rf4kc"]
Oct 30 20:27:21.603877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.603848    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:27:21.603999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.603895    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 20:27:21.964855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.964823    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc" path="/var/lib/kubelet/pods/4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc/volumes"
Oct 30 20:27:21.965143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.965130    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c624afa-0fdb-4725-b66e-419aa3cb0eab" path="/var/lib/kubelet/pods/6c624afa-0fdb-4725-b66e-419aa3cb0eab/volumes"
Oct 30 20:27:21.965417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:21.965406    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22fd554-ec00-49cd-893b-6c18ac83aaae" path="/var/lib/kubelet/pods/c22fd554-ec00-49cd-893b-6c18ac83aaae/volumes"
Oct 30 20:27:22.604095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:22.604059    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:27:22.604481 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:22.604105    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 20:27:23.604088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:23.604051    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:27:23.604244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:23.604106    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Oct 30 20:27:24.606869 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:24.606837    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:27:24.607346 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:24.607239    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:27:24.607837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:24.607819    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:27:26.792512 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:26.792478    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:32.232344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:27:32.232306    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:37.113256 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:27:37.113222    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:28:07.112798 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:28:07.112716    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:28:27.017214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:28:27.017140    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:28:37.113489 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:28:37.113447    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:28:56.418065 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:28:56.417997    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:28:56.604749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:28:56.604695    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:07.113384 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:29:07.113347    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:29:10.027693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:10.027654    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:17.167437 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167408    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"]
Oct 30 20:29:17.167820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167758    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc" containerName="authorino"
Oct 30 20:29:17.167820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167797    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc" containerName="authorino"
Oct 30 20:29:17.167820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167812    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c22fd554-ec00-49cd-893b-6c18ac83aaae" containerName="authorino"
Oct 30 20:29:17.167820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167818    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22fd554-ec00-49cd-893b-6c18ac83aaae" containerName="authorino"
Oct 30 20:29:17.167950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167825    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6c624afa-0fdb-4725-b66e-419aa3cb0eab" containerName="authorino"
Oct 30 20:29:17.167950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167830    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c624afa-0fdb-4725-b66e-419aa3cb0eab" containerName="authorino"
Oct 30 20:29:17.167950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167891    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8f9868-aeb7-4d42-b0e9-ae8e96572cdc" containerName="authorino"
Oct 30 20:29:17.167950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167899    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22fd554-ec00-49cd-893b-6c18ac83aaae" containerName="authorino"
Oct 30 20:29:17.167950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.167905    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c624afa-0fdb-4725-b66e-419aa3cb0eab" containerName="authorino"
Oct 30 20:29:17.170780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.170766    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.224036 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.224011    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 20:29:17.274508 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.274484    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 20:29:17.274646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.274561    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m9jq4\""
Oct 30 20:29:17.274646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.274559    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 20:29:17.276467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.276449    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"]
Oct 30 20:29:17.311739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.311712    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:17.335250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.335221    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389982d5-4a66-44c1-bb59-e293269b1e58-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.335391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.335253    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pk9\" (UniqueName: \"kubernetes.io/projected/389982d5-4a66-44c1-bb59-e293269b1e58-kube-api-access-v9pk9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.335391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.335352    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.335518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.335478    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.335518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.335508    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.335616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.335587    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.436885 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.436831    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.436885 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.436869    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.436885 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.436885    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.437091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.436904    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.437091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.436937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389982d5-4a66-44c1-bb59-e293269b1e58-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.437091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.436961    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pk9\" (UniqueName: \"kubernetes.io/projected/389982d5-4a66-44c1-bb59-e293269b1e58-kube-api-access-v9pk9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.437277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.437259    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.437324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.437303    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.437385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.437364    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.439321 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.439298    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389982d5-4a66-44c1-bb59-e293269b1e58-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.439574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.439557    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389982d5-4a66-44c1-bb59-e293269b1e58-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.497605 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.497587    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pk9\" (UniqueName: \"kubernetes.io/projected/389982d5-4a66-44c1-bb59-e293269b1e58-kube-api-access-v9pk9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fkjnq\" (UID: \"389982d5-4a66-44c1-bb59-e293269b1e58\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:17.782940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:17.782909    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:18.155264 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:29:18.155233    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389982d5_4a66_44c1_bb59_e293269b1e58.slice/crio-c0b98c4893d012a1431470ad1b093247c0c2dd2c06208a7dffc945915c3b05ef WatchSource:0}: Error finding container c0b98c4893d012a1431470ad1b093247c0c2dd2c06208a7dffc945915c3b05ef: Status 404 returned error can't find the container with id c0b98c4893d012a1431470ad1b093247c0c2dd2c06208a7dffc945915c3b05ef
Oct 30 20:29:18.155399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:18.155376    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"]
Oct 30 20:29:18.792802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:18.792769    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq" event={"ID":"389982d5-4a66-44c1-bb59-e293269b1e58","Type":"ContainerStarted","Data":"c0b98c4893d012a1431470ad1b093247c0c2dd2c06208a7dffc945915c3b05ef"}
Oct 30 20:29:19.271940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:19.271229    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:29:20.805923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:20.805889    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq" event={"ID":"389982d5-4a66-44c1-bb59-e293269b1e58","Type":"ContainerStarted","Data":"15a7c8003df2b267c42ce45842f4dacc69c57faec142bcaa5cfe67c947d7dd98"}
Oct 30 20:29:20.928213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:20.928124    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq" podStartSLOduration=1.524788312 podStartE2EDuration="3.928106709s" totalImagesPullingTime="2.403318397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:20.925194812 +0000 UTC m=+587.074245800" watchObservedRunningTime="2026-10-30 20:29:20.928106709 +0000 UTC m=+587.077157700"
Oct 30 20:29:21.178956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.178888    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"]
Oct 30 20:29:21.182413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.182398    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.238300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.238274    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 20:29:21.241141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.241120    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"]
Oct 30 20:29:21.269813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.269794    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947c828e-534f-4ec9-b78a-2c354191c54a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.269923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.269831    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.269923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.269860    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.270025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.269937    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.270025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.269967    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9v2v\" (UniqueName: \"kubernetes.io/projected/947c828e-534f-4ec9-b78a-2c354191c54a-kube-api-access-x9v2v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.270133 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.270059    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.272602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.272581    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:21.371421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371395    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.371575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371429    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.371575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371448    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.371711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371576    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9v2v\" (UniqueName: \"kubernetes.io/projected/947c828e-534f-4ec9-b78a-2c354191c54a-kube-api-access-x9v2v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.371711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371611    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.371711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371684    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947c828e-534f-4ec9-b78a-2c354191c54a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.371924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371804    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.371924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.371864    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.372035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.372017    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.373851 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.373824    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947c828e-534f-4ec9-b78a-2c354191c54a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.374081 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.374065    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947c828e-534f-4ec9-b78a-2c354191c54a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.429297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.429245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9v2v\" (UniqueName: \"kubernetes.io/projected/947c828e-534f-4ec9-b78a-2c354191c54a-kube-api-access-x9v2v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6\" (UID: \"947c828e-534f-4ec9-b78a-2c354191c54a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.494639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.494619    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:21.783132 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.783105    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:21.864092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:21.864066    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"]
Oct 30 20:29:21.865094 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:29:21.865071    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947c828e_534f_4ec9_b78a_2c354191c54a.slice/crio-3bb6450b38b35fdf60a327cd0ebb0376b63862f19a88997ae6d459435d05cb87 WatchSource:0}: Error finding container 3bb6450b38b35fdf60a327cd0ebb0376b63862f19a88997ae6d459435d05cb87: Status 404 returned error can't find the container with id 3bb6450b38b35fdf60a327cd0ebb0376b63862f19a88997ae6d459435d05cb87
Oct 30 20:29:22.816541 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:22.816492    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6" event={"ID":"947c828e-534f-4ec9-b78a-2c354191c54a","Type":"ContainerStarted","Data":"a2d362fd3fa09820dc7e87541626b4eb8a0656ecdefbd275ab8adf05ba29fe30"}
Oct 30 20:29:22.816692 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:22.816577    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6" event={"ID":"947c828e-534f-4ec9-b78a-2c354191c54a","Type":"ContainerStarted","Data":"3bb6450b38b35fdf60a327cd0ebb0376b63862f19a88997ae6d459435d05cb87"}
Oct 30 20:29:22.934932 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:22.934856    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6" podStartSLOduration=1.182219991 podStartE2EDuration="1.934842127s" totalImagesPullingTime="752.622136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:22.933685482 +0000 UTC m=+589.082736460" watchObservedRunningTime="2026-10-30 20:29:22.934842127 +0000 UTC m=+589.083893113"
Oct 30 20:29:23.494889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:23.494859    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:24.704286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.704254    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-58nn6"]
Oct 30 20:29:24.707707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.707692    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.801007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.800981    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.801164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.801012    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.801164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.801029    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g2l\" (UniqueName: \"kubernetes.io/projected/7b3937d6-0aa3-4da5-a364-5aff445cf25a-kube-api-access-74g2l\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.801164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.801109    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3937d6-0aa3-4da5-a364-5aff445cf25a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.801259 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.801241    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.801296 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.801271    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-home\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.810954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.810934    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 20:29:24.812944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.812926    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-58nn6"]
Oct 30 20:29:24.852835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.852810    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:24.902155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902126    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902163    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902185    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g2l\" (UniqueName: \"kubernetes.io/projected/7b3937d6-0aa3-4da5-a364-5aff445cf25a-kube-api-access-74g2l\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902243    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3937d6-0aa3-4da5-a364-5aff445cf25a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902282    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902315    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-home\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902613 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902589    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902698    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-home\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.902716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.902709    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.904421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.904404    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3937d6-0aa3-4da5-a364-5aff445cf25a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.904774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.904759    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3937d6-0aa3-4da5-a364-5aff445cf25a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:24.962108 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:24.962079    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g2l\" (UniqueName: \"kubernetes.io/projected/7b3937d6-0aa3-4da5-a364-5aff445cf25a-kube-api-access-74g2l\") pod \"echo-simulator-kserve-64b6f8f96c-58nn6\" (UID: \"7b3937d6-0aa3-4da5-a364-5aff445cf25a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:25.021617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:25.021598    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:25.393832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:25.393811    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-58nn6"]
Oct 30 20:29:25.395679 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:29:25.395648    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3937d6_0aa3_4da5_a364_5aff445cf25a.slice/crio-2411bda14b4642fc90b95aad1ab9d9dabe54897e52b0e3e7009348d5c7ac6e90 WatchSource:0}: Error finding container 2411bda14b4642fc90b95aad1ab9d9dabe54897e52b0e3e7009348d5c7ac6e90: Status 404 returned error can't find the container with id 2411bda14b4642fc90b95aad1ab9d9dabe54897e52b0e3e7009348d5c7ac6e90
Oct 30 20:29:25.837124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:25.837094    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6" event={"ID":"7b3937d6-0aa3-4da5-a364-5aff445cf25a","Type":"ContainerStarted","Data":"2411bda14b4642fc90b95aad1ab9d9dabe54897e52b0e3e7009348d5c7ac6e90"}
Oct 30 20:29:27.356842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.356812    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:27.638098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.637994    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"]
Oct 30 20:29:27.641878 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.641855    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.717399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.717368    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 20:29:27.727843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.727818    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a712478-adcf-48cc-a3ee-eb62d8e5460d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.727985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.727855    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.728048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.727995    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfsm\" (UniqueName: \"kubernetes.io/projected/8a712478-adcf-48cc-a3ee-eb62d8e5460d-kube-api-access-xrfsm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.728048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.728023    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.728125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.728053    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.728228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.728190    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.748182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.748148    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"]
Oct 30 20:29:27.770385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.770360    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:27.784854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.784830    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:27.799198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.799179    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:27.829300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829274    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829317    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a712478-adcf-48cc-a3ee-eb62d8e5460d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829346    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829420    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfsm\" (UniqueName: \"kubernetes.io/projected/8a712478-adcf-48cc-a3ee-eb62d8e5460d-kube-api-access-xrfsm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829462    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829497    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829709 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829690    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829793    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.829865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.829843    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.831620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.831601    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a712478-adcf-48cc-a3ee-eb62d8e5460d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.831865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.831849    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a712478-adcf-48cc-a3ee-eb62d8e5460d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.858446 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.858428    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fkjnq"
Oct 30 20:29:27.886877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.886859    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfsm\" (UniqueName: \"kubernetes.io/projected/8a712478-adcf-48cc-a3ee-eb62d8e5460d-kube-api-access-xrfsm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s\" (UID: \"8a712478-adcf-48cc-a3ee-eb62d8e5460d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:27.959073 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:27.959052    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:28.332940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:28.332908    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"]
Oct 30 20:29:28.383419 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:29:28.383395    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a712478_adcf_48cc_a3ee_eb62d8e5460d.slice/crio-f348d2e8ba519a8681d91966a3d3cef272878a4d0723aab6be6453092774fc4d WatchSource:0}: Error finding container f348d2e8ba519a8681d91966a3d3cef272878a4d0723aab6be6453092774fc4d: Status 404 returned error can't find the container with id f348d2e8ba519a8681d91966a3d3cef272878a4d0723aab6be6453092774fc4d
Oct 30 20:29:28.855558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:28.855513    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6" event={"ID":"7b3937d6-0aa3-4da5-a364-5aff445cf25a","Type":"ContainerStarted","Data":"96b9ce9769a899ed76e36abc0cfa9b13425924f8c24c5227901c891256cd39f3"}
Oct 30 20:29:28.857441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:28.856855    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s" event={"ID":"8a712478-adcf-48cc-a3ee-eb62d8e5460d","Type":"ContainerStarted","Data":"f348d2e8ba519a8681d91966a3d3cef272878a4d0723aab6be6453092774fc4d"}
Oct 30 20:29:28.975632 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:28.975590    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6" podStartSLOduration=2.498630215 podStartE2EDuration="4.975576722s" totalImagesPullingTime="2.476946507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:28.974014625 +0000 UTC m=+595.123065611" watchObservedRunningTime="2026-10-30 20:29:28.975576722 +0000 UTC m=+595.124627700"
Oct 30 20:29:29.022153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:29.022133    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:29.863857 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:29.863824    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s" event={"ID":"8a712478-adcf-48cc-a3ee-eb62d8e5460d","Type":"ContainerStarted","Data":"d09229d179946113ee2c2f4864c903e3b149e62cc473e5404d0566a1b40b14c2"}
Oct 30 20:29:29.964067 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:29.964042    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:29.982826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:29.982784    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s" podStartSLOduration=2.283565109 podStartE2EDuration="2.982771297s" totalImagesPullingTime="699.206188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:29.981367329 +0000 UTC m=+596.130418318" watchObservedRunningTime="2026-10-30 20:29:29.982771297 +0000 UTC m=+596.131822329"
Oct 30 20:29:30.084792 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:30.084768    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:29:30.944934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:30.944888    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:31.495861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:31.495835    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:31.507896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:31.507873    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:31.886279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:31.886212    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6"
Oct 30 20:29:34.052561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:34.052506    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:29:34.052994 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:34.052591    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:29:34.112361 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:34.112328    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:35.022704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:35.022675    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:35.034896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:35.034877    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:35.906103 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:35.906077    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-58nn6"
Oct 30 20:29:37.114274 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:29:37.114241    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:29:37.964095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:37.964067    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:37.978825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:37.978804    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:38.611731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.611691    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"]
Oct 30 20:29:38.617280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.617256    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.676823 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.676791    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 20:29:38.677009 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.676894    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"]
Oct 30 20:29:38.721264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.721230    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.721397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.721301    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7gb\" (UniqueName: \"kubernetes.io/projected/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-kube-api-access-4v7gb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.721477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.721453    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.721549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.721512    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.721611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.721589    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.721696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.721677    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.757010 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.756974    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:38.822827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.822794    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.822863    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7gb\" (UniqueName: \"kubernetes.io/projected/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-kube-api-access-4v7gb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.822929    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.822968    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.823000    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.823035    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.823298    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.823344    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.823991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.823607    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.826097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.826066    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.827026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.826971    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.883176 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.883116    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7gb\" (UniqueName: \"kubernetes.io/projected/8ca301d8-03b2-4873-a5f2-3f3dadf09b75-kube-api-access-4v7gb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5k9gz\" (UID: \"8ca301d8-03b2-4873-a5f2-3f3dadf09b75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:38.923561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.923522    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s"
Oct 30 20:29:38.931764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:38.931740    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:39.308764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:39.308746    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"]
Oct 30 20:29:39.310968 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:29:39.310937    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca301d8_03b2_4873_a5f2_3f3dadf09b75.slice/crio-3506dc85361e2fdc2f837271fdcf8fb45ba23cba68d787827f5a71bec86d7338 WatchSource:0}: Error finding container 3506dc85361e2fdc2f837271fdcf8fb45ba23cba68d787827f5a71bec86d7338: Status 404 returned error can't find the container with id 3506dc85361e2fdc2f837271fdcf8fb45ba23cba68d787827f5a71bec86d7338
Oct 30 20:29:39.919010 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:39.918977    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz" event={"ID":"8ca301d8-03b2-4873-a5f2-3f3dadf09b75","Type":"ContainerStarted","Data":"3506dc85361e2fdc2f837271fdcf8fb45ba23cba68d787827f5a71bec86d7338"}
Oct 30 20:29:40.926579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:40.926525    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz" event={"ID":"8ca301d8-03b2-4873-a5f2-3f3dadf09b75","Type":"ContainerStarted","Data":"d72f9c14074d56eae9593ad7bc9723d211c8490d7da3fa5f096a18959e1150ec"}
Oct 30 20:29:40.932875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:40.932857    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:41.042780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:41.042740    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz" podStartSLOduration=2.288732063 podStartE2EDuration="3.042723241s" totalImagesPullingTime="753.991178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:41.042215605 +0000 UTC m=+607.191266594" watchObservedRunningTime="2026-10-30 20:29:41.042723241 +0000 UTC m=+607.191774234"
Oct 30 20:29:42.955432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:42.955375    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:46.320105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:46.320074    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:29:48.932575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:48.932517    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:48.944445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:48.944426    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:48.985227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:48.985197    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5k9gz"
Oct 30 20:29:51.428025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:51.427994    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:52.793825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.793786    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"]
Oct 30 20:29:52.797837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.797815    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837161    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837191    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837232    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klh4\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-kube-api-access-6klh4\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837305    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837330    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837344    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837369    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837662 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837412    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.837662 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.837491    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.904831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.904798    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"]
Oct 30 20:29:52.938213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938180    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938213    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938236    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938276    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938297    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938452    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938521    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938563    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938751 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938640    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klh4\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-kube-api-access-6klh4\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.938853 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.938831    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.939130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.939102    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.939269 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.939188    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.939329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.939267    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.939329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.939273    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.940744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.940725    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.941347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.941332    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.995336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.995309    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:52.995761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:52.995744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klh4\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-kube-api-access-6klh4\") pod \"maas-default-gateway-openshift-default-6886b96f9-8l6hk\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:53.111781 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:53.111731    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:53.535271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:53.535243    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"]
Oct 30 20:29:53.585350 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:29:53.585312    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bba3496_9380_4fd4_b133_0dd4c4fc48b8.slice/crio-e80c1c3f4ad3daf9555e850200008a894f929c083d127bc03bb91c42e671638f WatchSource:0}: Error finding container e80c1c3f4ad3daf9555e850200008a894f929c083d127bc03bb91c42e671638f: Status 404 returned error can't find the container with id e80c1c3f4ad3daf9555e850200008a894f929c083d127bc03bb91c42e671638f
Oct 30 20:29:53.587385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:53.587342    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:29:53.587469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:53.587437    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:29:53.587579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:53.587481    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:29:54.000588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:54.000553    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" event={"ID":"6bba3496-9380-4fd4-b133-0dd4c4fc48b8","Type":"ContainerStarted","Data":"b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372"}
Oct 30 20:29:54.001069 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:54.000618    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" event={"ID":"6bba3496-9380-4fd4-b133-0dd4c4fc48b8","Type":"ContainerStarted","Data":"e80c1c3f4ad3daf9555e850200008a894f929c083d127bc03bb91c42e671638f"}
Oct 30 20:29:54.116204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:54.112668    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:29:54.119220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:54.118355    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:29:54.119220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:54.118423    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 20:29:55.127559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:55.125808    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:29:55.127559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:55.125862    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 20:29:56.121900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:56.121869    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:29:56.122174 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:56.122144    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 20:29:57.121863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:57.121828    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:29:57.122440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:57.122409    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 20:29:58.123557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:58.117816    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:29:58.123557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:58.117873    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 20:29:59.130557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.125806    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:29:59.130557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.125861    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 20:29:59.636899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.636838    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podStartSLOduration=7.636824161 podStartE2EDuration="7.636824161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:54.119650284 +0000 UTC m=+620.268701271" watchObservedRunningTime="2026-10-30 20:29:59.636824161 +0000 UTC m=+625.785875151"
Oct 30 20:29:59.638487 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.638457    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65f74f87f7-vvgwg"]
Oct 30 20:29:59.646466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.646451    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:29:59.698278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.698255    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f74f87f7-vvgwg"]
Oct 30 20:29:59.701279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.701253    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b9a79-f518-4135-af18-337625525bc4-tls-cert\") pod \"authorino-65f74f87f7-vvgwg\" (UID: \"467b9a79-f518-4135-af18-337625525bc4\") " pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:29:59.701388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.701311    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2q9\" (UniqueName: \"kubernetes.io/projected/467b9a79-f518-4135-af18-337625525bc4-kube-api-access-pn2q9\") pod \"authorino-65f74f87f7-vvgwg\" (UID: \"467b9a79-f518-4135-af18-337625525bc4\") " pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:29:59.746143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.746119    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 20:29:59.749462 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.749444    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2p4vw\""
Oct 30 20:29:59.802467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.802444    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b9a79-f518-4135-af18-337625525bc4-tls-cert\") pod \"authorino-65f74f87f7-vvgwg\" (UID: \"467b9a79-f518-4135-af18-337625525bc4\") " pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:29:59.802635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.802494    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2q9\" (UniqueName: \"kubernetes.io/projected/467b9a79-f518-4135-af18-337625525bc4-kube-api-access-pn2q9\") pod \"authorino-65f74f87f7-vvgwg\" (UID: \"467b9a79-f518-4135-af18-337625525bc4\") " pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:29:59.805079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.805053    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/467b9a79-f518-4135-af18-337625525bc4-tls-cert\") pod \"authorino-65f74f87f7-vvgwg\" (UID: \"467b9a79-f518-4135-af18-337625525bc4\") " pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:29:59.862453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.862425    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2q9\" (UniqueName: \"kubernetes.io/projected/467b9a79-f518-4135-af18-337625525bc4-kube-api-access-pn2q9\") pod \"authorino-65f74f87f7-vvgwg\" (UID: \"467b9a79-f518-4135-af18-337625525bc4\") " pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:29:59.931379 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.931303    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:59.974009 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:29:59.973979    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f74f87f7-vvgwg"
Oct 30 20:30:00.104076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:00.104047    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:00.112345 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:00.112318    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:30:00.112472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:00.112368    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 20:30:00.353866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:00.353844    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f74f87f7-vvgwg"]
Oct 30 20:30:00.355951 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:30:00.355919    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod467b9a79_f518_4135_af18_337625525bc4.slice/crio-11560f90875f8f35f9691b42d13f60f63d9d57bbda7cdc5b93fdbee110799473 WatchSource:0}: Error finding container 11560f90875f8f35f9691b42d13f60f63d9d57bbda7cdc5b93fdbee110799473: Status 404 returned error can't find the container with id 11560f90875f8f35f9691b42d13f60f63d9d57bbda7cdc5b93fdbee110799473
Oct 30 20:30:00.357467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:00.357438    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:30:01.063428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:01.063384    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f74f87f7-vvgwg" event={"ID":"467b9a79-f518-4135-af18-337625525bc4","Type":"ContainerStarted","Data":"11560f90875f8f35f9691b42d13f60f63d9d57bbda7cdc5b93fdbee110799473"}
Oct 30 20:30:01.615981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:01.615949    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:30:01.616921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:01.616902    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:30:01.617820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:01.617803    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:30:03.076562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:03.076505    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f74f87f7-vvgwg" event={"ID":"467b9a79-f518-4135-af18-337625525bc4","Type":"ContainerStarted","Data":"4620f18904b2f3ce80f0d8fd8bb41d7f0df25e6b5813895accdc647949ac5526"}
Oct 30 20:30:03.208274 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:03.208221    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65f74f87f7-vvgwg" podStartSLOduration=2.48652554 podStartE2EDuration="4.208205919s" totalImagesPullingTime="1.721680379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.205793521 +0000 UTC m=+629.354844508" watchObservedRunningTime="2026-10-30 20:30:03.208205919 +0000 UTC m=+629.357256906"
Oct 30 20:30:07.117451 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:30:07.117410    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:30:07.977549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:07.977502    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:08.160107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:08.160074    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:10.529787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:10.529751    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:10.703315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:10.703286    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:27.697200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:27.697168    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:33.156142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:33.156086    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:33.678492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:33.678459    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:35.375372 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:35.375335    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:35.567818 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:35.567789    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:36.994179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:36.994144    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:37.113766 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:30:37.113734    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:30:37.187842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:37.187812    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:53.949862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:53.949779    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:54.123486 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:54.123450    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:57.944366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:57.944335    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:58.234117 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:30:58.234092    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:03.701214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:03.701183    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:03.896773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:03.896743    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:04.393157 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.393118    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:04.554355 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.554328    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"]
Oct 30 20:31:04.557961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.557944    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623131 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623106    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnhm\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-kube-api-access-rjnhm\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623276 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623145    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623276 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623163    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623322    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623356    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623489 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623454    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623527 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623508    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623631    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.623702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.623678    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.626279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.626257    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-4ee72d-openshift-default-dockercfg-5dldm\""
Oct 30 20:31:04.678515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.678457    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"]
Oct 30 20:31:04.699264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.699230    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:04.724332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724313    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724346    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724378    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724420    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnhm\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-kube-api-access-rjnhm\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724462    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724481    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724523    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724574    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.724711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724626    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.725055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724762    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.725055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.724927    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.725055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.725020    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.725055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.725040    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.725308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.725292    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.726739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.726723    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.727432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.727418    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.793260 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.793228    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnhm\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-kube-api-access-rjnhm\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.793850 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.793831    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.871767 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.871747    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:04.908235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:04.908208    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:05.136875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.136837    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:05.230230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.230205    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"]
Oct 30 20:31:05.234901 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.234884    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.299862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.299839    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ae2d38-openshift-default-dockercfg-9z4cf\""
Oct 30 20:31:05.309548 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.309503    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"]
Oct 30 20:31:05.329968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.329938    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.329980    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330118 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.330074    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330118 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.330099    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.330128    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.330213    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.330241    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330293 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.330280    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwfr\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-kube-api-access-xqwfr\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.330365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.330357    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.380502 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:05.380185    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92c5d8c_4387_4c48_ad67_a6a6d03d2ef2.slice/crio-8f3ff7b32e8ebff61fe05aa9d3fd49642932f0eda2a0e908ee46766a9470e084 WatchSource:0}: Error finding container 8f3ff7b32e8ebff61fe05aa9d3fd49642932f0eda2a0e908ee46766a9470e084: Status 404 returned error can't find the container with id 8f3ff7b32e8ebff61fe05aa9d3fd49642932f0eda2a0e908ee46766a9470e084
Oct 30 20:31:05.381952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.381916    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:05.382038 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.381996    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:05.382082 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.382038    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:05.382677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.382659    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"]
Oct 30 20:31:05.423598 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.423565    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" event={"ID":"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2","Type":"ContainerStarted","Data":"8f3ff7b32e8ebff61fe05aa9d3fd49642932f0eda2a0e908ee46766a9470e084"}
Oct 30 20:31:05.430842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.430819    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.430951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.430854    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.430951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.430870    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.430951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.430904    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.430951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.430922    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.430951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.430945    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.430983    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.431014    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.431064    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwfr\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-kube-api-access-xqwfr\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431285 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.431261    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431348 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.431267    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.431558    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.431574    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.431763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.431718    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.433264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.433246    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.433379 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.433364    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.468523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.468501    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:05.499004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.498905    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwfr\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-kube-api-access-xqwfr\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.499282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.499266    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.548471 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.548443    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:05.795743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.795499    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:05.966540 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.966503    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"]
Oct 30 20:31:05.971249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.971233    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:05.979033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:05.979005    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"]
Oct 30 20:31:06.011014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.010893    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:06.013940 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:06.013914    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56204e2f_0989_4a1a_a7c2_dd9a707b827e.slice/crio-1870ef46952b44c09ce12600913ba90b3bd10f19c0d55c8fab632fae93720c5c WatchSource:0}: Error finding container 1870ef46952b44c09ce12600913ba90b3bd10f19c0d55c8fab632fae93720c5c: Status 404 returned error can't find the container with id 1870ef46952b44c09ce12600913ba90b3bd10f19c0d55c8fab632fae93720c5c
Oct 30 20:31:06.016291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.016263    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.016363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.016323    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.016363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.016351    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.033813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.029122    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"]
Oct 30 20:31:06.033813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.029403    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f9b95-openshift-default-dockercfg-kpgjw\""
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038619    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038683    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038712    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038736    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqf2\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-kube-api-access-9fqf2\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038784    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038813    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038846    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038880    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.039654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.038919    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140144    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140269    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140339    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140372    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqf2\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-kube-api-access-9fqf2\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140427    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140459    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140491    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140526    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140583    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.140675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140608    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.141046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.140933    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.142250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.141184    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.142250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.141476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.142250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.141869    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.144441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.144414    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.146289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.146266    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.199329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.199298    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.200176 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.200153    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqf2\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-kube-api-access-9fqf2\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.220313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.220290    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:06.286635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.286606    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:06.430611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.430573    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" event={"ID":"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2","Type":"ContainerStarted","Data":"4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b"}
Oct 30 20:31:06.432331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.432299    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" event={"ID":"56204e2f-0989-4a1a-a7c2-dd9a707b827e","Type":"ContainerStarted","Data":"b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c"}
Oct 30 20:31:06.432493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.432339    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" event={"ID":"56204e2f-0989-4a1a-a7c2-dd9a707b827e","Type":"ContainerStarted","Data":"1870ef46952b44c09ce12600913ba90b3bd10f19c0d55c8fab632fae93720c5c"}
Oct 30 20:31:06.461779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.461621    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:06.549600 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.549576    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:06.561221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.561133    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" podStartSLOduration=2.561119375 podStartE2EDuration="2.561119375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:06.558962643 +0000 UTC m=+692.708013631" watchObservedRunningTime="2026-10-30 20:31:06.561119375 +0000 UTC m=+692.710170362"
Oct 30 20:31:06.671439 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.671391    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:06.681388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.681347    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" podStartSLOduration=1.681333982 podStartE2EDuration="1.681333982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:06.679948053 +0000 UTC m=+692.828999040" watchObservedRunningTime="2026-10-30 20:31:06.681333982 +0000 UTC m=+692.830384969"
Oct 30 20:31:06.717495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.717470    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"]
Oct 30 20:31:06.733393 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:06.733370    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ad1cb3_708e_48f5_abe2_be61ea83e237.slice/crio-689311eb7584a5dc5cab2a1f7f080ddcae0b7baa20919de57f1fd4f0b9c36792 WatchSource:0}: Error finding container 689311eb7584a5dc5cab2a1f7f080ddcae0b7baa20919de57f1fd4f0b9c36792: Status 404 returned error can't find the container with id 689311eb7584a5dc5cab2a1f7f080ddcae0b7baa20919de57f1fd4f0b9c36792
Oct 30 20:31:06.736058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.736031    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.736118 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.736091    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.736161 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.736119    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.872635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.872548    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:06.877884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.877834    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:06.991115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:06.991080    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:07.116166 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:07.116133    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:31:07.438845 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:07.438803    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" event={"ID":"e6ad1cb3-708e-48f5-abe2-be61ea83e237","Type":"ContainerStarted","Data":"900d36b82d090acafa474456f7a3c91d019227a1a5a93853dc515666fb097353"}
Oct 30 20:31:07.439020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:07.438857    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" event={"ID":"e6ad1cb3-708e-48f5-abe2-be61ea83e237","Type":"ContainerStarted","Data":"689311eb7584a5dc5cab2a1f7f080ddcae0b7baa20919de57f1fd4f0b9c36792"}
Oct 30 20:31:07.439303 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:07.439287    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:07.440063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:07.440043    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:07.553981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:07.553954    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:07.561843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:07.561802    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podStartSLOduration=2.561789907 podStartE2EDuration="2.561789907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:07.559390991 +0000 UTC m=+693.708441979" watchObservedRunningTime="2026-10-30 20:31:07.561789907 +0000 UTC m=+693.710840892"
Oct 30 20:31:08.287450 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:08.287419    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:08.292647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:08.292623    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:08.444033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:08.444002    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:08.444193 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:08.444057    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:08.444701 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:08.444673    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:31:08.445019 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:08.445002    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:08.945400 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:08.945369    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:09.198304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:09.198236    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:10.042147 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:10.042112    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:10.240100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:10.240070    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:10.643093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:10.643065    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:10.854052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:10.854022    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:12.347405 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:12.347376    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:12.565644 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:12.565599    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:12.924930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:12.924894    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:13.146472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:13.146443    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:13.347588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:13.347556    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:13.682156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:13.682080    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:13.914764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:13.914733    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:15.539083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:15.539058    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:15.739777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:15.739752    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:18.048877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.048838    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:18.249912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.249860    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:18.326459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.326394    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"]
Oct 30 20:31:18.330015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.329997    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:18.439473 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.439444    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-4ee72d\""
Oct 30 20:31:18.440588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.440569    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-4ee72d-dockercfg-mn8rs\""
Oct 30 20:31:18.441928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.441909    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"]
Oct 30 20:31:18.457883 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.457860    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9fa523-13bb-4f20-9410-b4e49f214dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:18.457975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.457907    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vkj\" (UniqueName: \"kubernetes.io/projected/bc9fa523-13bb-4f20-9410-b4e49f214dd1-kube-api-access-k6vkj\") pod \"payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:18.559113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.559082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9fa523-13bb-4f20-9410-b4e49f214dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:18.559269 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.559132    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vkj\" (UniqueName: \"kubernetes.io/projected/bc9fa523-13bb-4f20-9410-b4e49f214dd1-kube-api-access-k6vkj\") pod \"payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:18.559984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.559960    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9fa523-13bb-4f20-9410-b4e49f214dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:18.619664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.619607    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vkj\" (UniqueName: \"kubernetes.io/projected/bc9fa523-13bb-4f20-9410-b4e49f214dd1-kube-api-access-k6vkj\") pod \"payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:18.643547 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:18.643507    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:19.029333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:19.029307    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"]
Oct 30 20:31:19.029820 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:19.029795    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9fa523_13bb_4f20_9410_b4e49f214dd1.slice/crio-d7bce760c292f67d72543e843ceaee2ab0e5fc12af07870923f19405221c8b59 WatchSource:0}: Error finding container d7bce760c292f67d72543e843ceaee2ab0e5fc12af07870923f19405221c8b59: Status 404 returned error can't find the container with id d7bce760c292f67d72543e843ceaee2ab0e5fc12af07870923f19405221c8b59
Oct 30 20:31:19.515408 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:19.515379    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg" event={"ID":"bc9fa523-13bb-4f20-9410-b4e49f214dd1","Type":"ContainerStarted","Data":"d7bce760c292f67d72543e843ceaee2ab0e5fc12af07870923f19405221c8b59"}
Oct 30 20:31:20.504288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.504253    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"]
Oct 30 20:31:20.508296 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.508274    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.512893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.512866    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:20.580091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.580055    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-metrics-certs\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.580668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.580124    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/991c5524-1638-4f50-ab27-289d7ab6824a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.580668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.580162    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzf4t\" (UniqueName: \"kubernetes.io/projected/991c5524-1638-4f50-ab27-289d7ab6824a-kube-api-access-vzf4t\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.580668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.580192    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.613757 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.613720    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:31:20.618336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.618309    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:31:20.618336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.618309    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:31:20.618562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.618366    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-0a78ac-serving-cert\""
Oct 30 20:31:20.618562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.618459    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xgxz5\""
Oct 30 20:31:20.622229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.622154    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"]
Oct 30 20:31:20.681216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.681187    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-metrics-certs\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.681365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.681272    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/991c5524-1638-4f50-ab27-289d7ab6824a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.681365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.681314    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzf4t\" (UniqueName: \"kubernetes.io/projected/991c5524-1638-4f50-ab27-289d7ab6824a-kube-api-access-vzf4t\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.681365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.681345    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.681584 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:20.681567    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-serving-cert: secret "maas-api-e2e-worker-w0-0a78ac-serving-cert" not found
Oct 30 20:31:20.681643 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:20.681622    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls podName:991c5524-1638-4f50-ab27-289d7ab6824a nodeName:}" failed. No retries permitted until 2026-10-30 20:31:21.181605533 +0000 UTC m=+707.330656499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls") pod "maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc" (UID: "991c5524-1638-4f50-ab27-289d7ab6824a") : secret "maas-api-e2e-worker-w0-0a78ac-serving-cert" not found
Oct 30 20:31:20.681983 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.681964    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/991c5524-1638-4f50-ab27-289d7ab6824a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.683843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.683820    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-metrics-certs\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.700868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.700840    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:20.742025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.742005    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzf4t\" (UniqueName: \"kubernetes.io/projected/991c5524-1638-4f50-ab27-289d7ab6824a-kube-api-access-vzf4t\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:20.891151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:20.891070    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:21.079973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:21.079938    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:21.187997 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:21.187910    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:21.190848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:21.190823    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:21.434183 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:21.434149    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:21.586021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:21.585988    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:21.796030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:21.796000    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:21.828171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:21.828141    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"]
Oct 30 20:31:21.828876 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:21.828848    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991c5524_1638_4f50_ab27_289d7ab6824a.slice/crio-cfeee41886717f3d40a88614e841d09a7aed1360f0ce3b6dffa4443904b713d1 WatchSource:0}: Error finding container cfeee41886717f3d40a88614e841d09a7aed1360f0ce3b6dffa4443904b713d1: Status 404 returned error can't find the container with id cfeee41886717f3d40a88614e841d09a7aed1360f0ce3b6dffa4443904b713d1
Oct 30 20:31:22.533919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:22.533887    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg" event={"ID":"bc9fa523-13bb-4f20-9410-b4e49f214dd1","Type":"ContainerStarted","Data":"f3734afcbee7ece60bc41d19023a77c752f1e418da8782b52414e03717bdf37f"}
Oct 30 20:31:22.534113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:22.534022    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:22.535193 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:22.535171    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc" event={"ID":"991c5524-1638-4f50-ab27-289d7ab6824a","Type":"ContainerStarted","Data":"cfeee41886717f3d40a88614e841d09a7aed1360f0ce3b6dffa4443904b713d1"}
Oct 30 20:31:22.651148 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:22.651106    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg" podStartSLOduration=1.782281281 podStartE2EDuration="4.651094256s" totalImagesPullingTime="2.868812975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:22.649621197 +0000 UTC m=+708.798672185" watchObservedRunningTime="2026-10-30 20:31:22.651094256 +0000 UTC m=+708.800145242"
Oct 30 20:31:23.860214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:23.860181    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:31:25.440064 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.440022    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:25.636938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.636904    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:25.805982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.805962    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"]
Oct 30 20:31:25.835881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.835838    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:25.870003 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.869980    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"]
Oct 30 20:31:25.939045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.939015    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-a78bf4\""
Oct 30 20:31:25.968778 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.964993    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0423fc-58b9-42f3-941c-f664d7646730-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:25.968778 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.965110    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbfm\" (UniqueName: \"kubernetes.io/projected/0d0423fc-58b9-42f3-941c-f664d7646730-kube-api-access-kcbfm\") pod \"payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:25.968778 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.967441    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"]
Oct 30 20:31:25.971233 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.971218    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:25.990519 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:25.990486    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-a78bf4-dockercfg-4mbqh\""
Oct 30 20:31:26.054691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.053582    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"]
Oct 30 20:31:26.066912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.066425    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbfm\" (UniqueName: \"kubernetes.io/projected/0d0423fc-58b9-42f3-941c-f664d7646730-kube-api-access-kcbfm\") pod \"payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:26.066912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.066476    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xnb\" (UniqueName: \"kubernetes.io/projected/81a9e296-a407-4ef0-878b-5c10dea1613f-kube-api-access-97xnb\") pod \"payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:26.067159 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.066527    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0423fc-58b9-42f3-941c-f664d7646730-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:26.067987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.067337    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a9e296-a407-4ef0-878b-5c10dea1613f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:26.067987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.067435    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0423fc-58b9-42f3-941c-f664d7646730-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:26.073875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.073617    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:26.138349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.138315    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbfm\" (UniqueName: \"kubernetes.io/projected/0d0423fc-58b9-42f3-941c-f664d7646730-kube-api-access-kcbfm\") pod \"payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:26.149998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.149971    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:26.168346 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.168322    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a9e296-a407-4ef0-878b-5c10dea1613f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:26.168451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.168360    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xnb\" (UniqueName: \"kubernetes.io/projected/81a9e296-a407-4ef0-878b-5c10dea1613f-kube-api-access-97xnb\") pod \"payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:26.169021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.169003    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a9e296-a407-4ef0-878b-5c10dea1613f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:26.255779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.255630    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:26.267295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.265164    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xnb\" (UniqueName: \"kubernetes.io/projected/81a9e296-a407-4ef0-878b-5c10dea1613f-kube-api-access-97xnb\") pod \"payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:26.285076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.285046    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:26.535304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.535270    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"]
Oct 30 20:31:26.536818 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:26.536777    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0423fc_58b9_42f3_941c_f664d7646730.slice/crio-5403f309efb2d49cbfe83aac009857a5bc8c5bb7f456e22bb878a346bc3bcb92 WatchSource:0}: Error finding container 5403f309efb2d49cbfe83aac009857a5bc8c5bb7f456e22bb878a346bc3bcb92: Status 404 returned error can't find the container with id 5403f309efb2d49cbfe83aac009857a5bc8c5bb7f456e22bb878a346bc3bcb92
Oct 30 20:31:26.564653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.564622    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc" event={"ID":"991c5524-1638-4f50-ab27-289d7ab6824a","Type":"ContainerStarted","Data":"887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f"}
Oct 30 20:31:26.564772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.564693    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:26.565748 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.565726    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" event={"ID":"0d0423fc-58b9-42f3-941c-f664d7646730","Type":"ContainerStarted","Data":"5403f309efb2d49cbfe83aac009857a5bc8c5bb7f456e22bb878a346bc3bcb92"}
Oct 30 20:31:26.687080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.687056    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"]
Oct 30 20:31:26.688670 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:26.688645    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a9e296_a407_4ef0_878b_5c10dea1613f.slice/crio-16a18990ce1703771285ac82c8a830d3f8433b3577bf5009dc517940e75badc4 WatchSource:0}: Error finding container 16a18990ce1703771285ac82c8a830d3f8433b3577bf5009dc517940e75badc4: Status 404 returned error can't find the container with id 16a18990ce1703771285ac82c8a830d3f8433b3577bf5009dc517940e75badc4
Oct 30 20:31:26.696679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:26.696637    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc" podStartSLOduration=2.770832388 podStartE2EDuration="6.696622846s" totalImagesPullingTime="3.925790458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:26.695099956 +0000 UTC m=+712.844150944" watchObservedRunningTime="2026-10-30 20:31:26.696622846 +0000 UTC m=+712.845673865"
Oct 30 20:31:27.573411 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:27.573377    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" event={"ID":"81a9e296-a407-4ef0-878b-5c10dea1613f","Type":"ContainerStarted","Data":"37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7"}
Oct 30 20:31:27.573938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:27.573428    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" event={"ID":"81a9e296-a407-4ef0-878b-5c10dea1613f","Type":"ContainerStarted","Data":"16a18990ce1703771285ac82c8a830d3f8433b3577bf5009dc517940e75badc4"}
Oct 30 20:31:27.573938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:27.573489    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:27.575313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:27.575282    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" event={"ID":"0d0423fc-58b9-42f3-941c-f664d7646730","Type":"ContainerStarted","Data":"290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e"}
Oct 30 20:31:27.575471 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:27.575357    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:27.694516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:27.694457    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" podStartSLOduration=2.69444538 podStartE2EDuration="2.69444538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:27.691923072 +0000 UTC m=+713.840974060" watchObservedRunningTime="2026-10-30 20:31:27.69444538 +0000 UTC m=+713.843496366"
Oct 30 20:31:27.813558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:27.809754    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" podStartSLOduration=2.809736934 podStartE2EDuration="2.809736934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:27.807260687 +0000 UTC m=+713.956311673" watchObservedRunningTime="2026-10-30 20:31:27.809736934 +0000 UTC m=+713.958787922"
Oct 30 20:31:28.036612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:28.036575    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:28.268879 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:28.268846    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:30.874742 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:30.874699    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:31.070360 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:31.070328    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:32.581107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:32.581081    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 20:31:32.933699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:32.933625    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:31:33.547595 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:33.547568    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 20:31:33.624841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:33.624809    5002 generic.go:356] "Generic (PLEG): container finished" podID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerID="4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b" exitCode=0
Oct 30 20:31:33.625373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:33.624889    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" event={"ID":"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2","Type":"ContainerDied","Data":"4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b"}
Oct 30 20:31:33.625373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:33.625295    5002 scope.go:122] "RemoveContainer" containerID="4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b"
Oct 30 20:31:33.626939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:33.626908    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:33.627059 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:33.626976    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:33.627059 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:33.627008    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:34.650558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:34.650072    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" event={"ID":"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2","Type":"ContainerStarted","Data":"1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0"}
Oct 30 20:31:34.874574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:34.873692    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:34.874574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:34.873962    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:31:34.874574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:34.874017    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 20:31:35.275013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.274977    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:35.471559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.466875    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:35.639989 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.639903    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"]
Oct 30 20:31:35.645440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.645421    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:35.682905 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.681592    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2290d2df-e80e-4348-93cb-8a09f6357953-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:35.682905 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.681661    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9mq\" (UniqueName: \"kubernetes.io/projected/2290d2df-e80e-4348-93cb-8a09f6357953-kube-api-access-lr9mq\") pod \"payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:35.685547 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.685507    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"]
Oct 30 20:31:35.722556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.721940    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8e2fc1\""
Oct 30 20:31:35.731179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.730014    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8e2fc1-dockercfg-65tpx\""
Oct 30 20:31:35.782813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.782757    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2290d2df-e80e-4348-93cb-8a09f6357953-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:35.783017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.782831    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9mq\" (UniqueName: \"kubernetes.io/projected/2290d2df-e80e-4348-93cb-8a09f6357953-kube-api-access-lr9mq\") pod \"payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:35.783816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.783766    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2290d2df-e80e-4348-93cb-8a09f6357953-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:35.873076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.873038    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9mq\" (UniqueName: \"kubernetes.io/projected/2290d2df-e80e-4348-93cb-8a09f6357953-kube-api-access-lr9mq\") pod \"payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:35.878987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.878627    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:31:35.878987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.878677    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 20:31:35.879992 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.879697    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:35.973852 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:35.973830    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:36.400639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:36.400608    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"]
Oct 30 20:31:36.401905 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:36.401873    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2290d2df_e80e_4348_93cb_8a09f6357953.slice/crio-018bac28dae9a8f9d7159bc4f277a8591f07c13bb3a8492502b8e511ead3a3ee WatchSource:0}: Error finding container 018bac28dae9a8f9d7159bc4f277a8591f07c13bb3a8492502b8e511ead3a3ee: Status 404 returned error can't find the container with id 018bac28dae9a8f9d7159bc4f277a8591f07c13bb3a8492502b8e511ead3a3ee
Oct 30 20:31:36.674493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:36.674408    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" event={"ID":"2290d2df-e80e-4348-93cb-8a09f6357953","Type":"ContainerStarted","Data":"a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244"}
Oct 30 20:31:36.674493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:36.674462    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" event={"ID":"2290d2df-e80e-4348-93cb-8a09f6357953","Type":"ContainerStarted","Data":"018bac28dae9a8f9d7159bc4f277a8591f07c13bb3a8492502b8e511ead3a3ee"}
Oct 30 20:31:36.674735 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:36.674715    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:36.798619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:36.798556    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" podStartSLOduration=1.7985271680000001 podStartE2EDuration="1.798527168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:36.797526277 +0000 UTC m=+722.946577266" watchObservedRunningTime="2026-10-30 20:31:36.798527168 +0000 UTC m=+722.947578149"
Oct 30 20:31:36.872278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:36.872239    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:31:36.872452 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:36.872288    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 20:31:37.158569 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:37.153723    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:31:37.440611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:37.439633    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:37.837557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:37.833913    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:37.873020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:37.872977    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:31:37.873197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:37.873038    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 20:31:38.138238 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.138128    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:38.235865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.235825    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"]
Oct 30 20:31:38.245829 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.245804    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.286593 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.286567    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"]
Oct 30 20:31:38.341681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.341651    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ae2d38\""
Oct 30 20:31:38.359568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.342112    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ae2d38-dockercfg-f56p2\""
Oct 30 20:31:38.360133 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.360105    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7dq\" (UniqueName: \"kubernetes.io/projected/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-kube-api-access-bs7dq\") pod \"payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.360495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.360475    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.471635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.465313    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7dq\" (UniqueName: \"kubernetes.io/projected/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-kube-api-access-bs7dq\") pod \"payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.471635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.465405    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.471635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.466267    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.572244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.572212    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7dq\" (UniqueName: \"kubernetes.io/projected/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-kube-api-access-bs7dq\") pod \"payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.576561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.574926    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:38.587236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.587215    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:31:38.588279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.588258    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:31:38.873220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.873130    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:31:38.873220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.873196    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 20:31:38.996270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:38.996242    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"]
Oct 30 20:31:39.698865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:39.698834    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6" event={"ID":"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb","Type":"ContainerStarted","Data":"86a0d22c93b9808eeb81e7885342a4c7e6e6dc350b876597d05d122f77a931e3"}
Oct 30 20:31:39.699058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:39.698884    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6" event={"ID":"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb","Type":"ContainerStarted","Data":"14395a9b1b39f971ac71217cd5d99fad03e5c2c9a012dd8a12edcf647ed3b65e"}
Oct 30 20:31:39.699058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:39.699020    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:39.814799 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:39.814707    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6" podStartSLOduration=1.814691953 podStartE2EDuration="1.814691953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:39.814106205 +0000 UTC m=+725.963157183" watchObservedRunningTime="2026-10-30 20:31:39.814691953 +0000 UTC m=+725.963742934"
Oct 30 20:31:39.876034 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:39.876007    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:39.877346 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:39.877316    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 20:31:40.294510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.294477    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"]
Oct 30 20:31:40.299045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.299016    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.352697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.352653    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-a78bf4-serving-cert\""
Oct 30 20:31:40.356740 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.356709    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"]
Oct 30 20:31:40.391385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.391354    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.391612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.391394    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-metrics-certs\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.391612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.391423    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhwx\" (UniqueName: \"kubernetes.io/projected/3032376b-74fb-403f-8e88-ea7d10c7421f-kube-api-access-9dhwx\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.391612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.391593    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3032376b-74fb-403f-8e88-ea7d10c7421f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.493158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.493118    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.493371 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.493176    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-metrics-certs\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.493371 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.493212    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhwx\" (UniqueName: \"kubernetes.io/projected/3032376b-74fb-403f-8e88-ea7d10c7421f-kube-api-access-9dhwx\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.493371 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.493286    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3032376b-74fb-403f-8e88-ea7d10c7421f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.493371 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:40.493291    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-serving-cert: secret "maas-api-e2e-worker-w5-a78bf4-serving-cert" not found
Oct 30 20:31:40.493625 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:40.493383    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls podName:3032376b-74fb-403f-8e88-ea7d10c7421f nodeName:}" failed. No retries permitted until 2026-10-30 20:31:40.993360389 +0000 UTC m=+727.142411357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls") pod "maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8" (UID: "3032376b-74fb-403f-8e88-ea7d10c7421f") : secret "maas-api-e2e-worker-w5-a78bf4-serving-cert" not found
Oct 30 20:31:40.494120 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.494099    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3032376b-74fb-403f-8e88-ea7d10c7421f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.495962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.495941    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-metrics-certs\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.534095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.534055    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:40.552740 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.552656    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhwx\" (UniqueName: \"kubernetes.io/projected/3032376b-74fb-403f-8e88-ea7d10c7421f-kube-api-access-9dhwx\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:40.711446 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.711415    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:40.884768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.884683    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:40.998076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:40.998040    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:41.000668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:41.000641    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:41.217888 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:41.217854    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:41.614686 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:41.614656    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"]
Oct 30 20:31:41.615309 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:41.615263    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3032376b_74fb_403f_8e88_ea7d10c7421f.slice/crio-9d760a279d75ff0afbe04d1e26528f37e2fd205f82d4decfaf8bdc18e68f73ea WatchSource:0}: Error finding container 9d760a279d75ff0afbe04d1e26528f37e2fd205f82d4decfaf8bdc18e68f73ea: Status 404 returned error can't find the container with id 9d760a279d75ff0afbe04d1e26528f37e2fd205f82d4decfaf8bdc18e68f73ea
Oct 30 20:31:41.722122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:41.722086    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8" event={"ID":"3032376b-74fb-403f-8e88-ea7d10c7421f","Type":"ContainerStarted","Data":"9d760a279d75ff0afbe04d1e26528f37e2fd205f82d4decfaf8bdc18e68f73ea"}
Oct 30 20:31:42.141259 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:42.141227    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:31:42.731410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:42.731381    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8" event={"ID":"3032376b-74fb-403f-8e88-ea7d10c7421f","Type":"ContainerStarted","Data":"3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa"}
Oct 30 20:31:42.732286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:42.731561    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:42.849685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:42.849641    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8" podStartSLOduration=2.027809797 podStartE2EDuration="2.849627846s" totalImagesPullingTime="821.818049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:42.849008669 +0000 UTC m=+728.998059656" watchObservedRunningTime="2026-10-30 20:31:42.849627846 +0000 UTC m=+728.998678833"
Oct 30 20:31:43.738923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:43.738891    5002 generic.go:356] "Generic (PLEG): container finished" podID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerID="b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c" exitCode=0
Oct 30 20:31:43.739321 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:43.738969    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" event={"ID":"56204e2f-0989-4a1a-a7c2-dd9a707b827e","Type":"ContainerDied","Data":"b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c"}
Oct 30 20:31:43.739366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:43.739347    5002 scope.go:122] "RemoveContainer" containerID="b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c"
Oct 30 20:31:43.740523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:43.740498    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:43.740612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:43.740586    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:43.740648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:43.740619    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:44.747971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:44.747937    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" event={"ID":"56204e2f-0989-4a1a-a7c2-dd9a707b827e","Type":"ContainerStarted","Data":"bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047"}
Oct 30 20:31:45.401094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:45.401062    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:45.549366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:45.549336    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:45.549693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:45.549669    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:31:45.549804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:45.549733    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 20:31:46.549735 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:46.549701    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:31:46.550142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:46.549763    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 20:31:47.549011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:47.548973    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:31:47.549187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:47.549040    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 20:31:47.684125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:47.684098    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 20:31:48.549494 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:48.549457    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:31:48.549707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:48.549519    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 20:31:48.744549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:48.744505    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:31:49.549874 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.549840    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:31:49.550124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.550097    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 20:31:49.802443 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.802362    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"]
Oct 30 20:31:49.806171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.806156    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:49.900506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.900468    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-metrics-certs\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:49.900783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.900527    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:49.900783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.900662    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7t6f\" (UniqueName: \"kubernetes.io/projected/f5002a04-b76d-44cf-8e5f-7063871fe6e1-kube-api-access-j7t6f\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:49.900783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.900764    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5002a04-b76d-44cf-8e5f-7063871fe6e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:49.910890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.910869    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5f9b95-serving-cert\""
Oct 30 20:31:49.914113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:49.914090    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"]
Oct 30 20:31:50.001731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.001696    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-metrics-certs\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.001900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.001737    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.001900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.001770    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7t6f\" (UniqueName: \"kubernetes.io/projected/f5002a04-b76d-44cf-8e5f-7063871fe6e1-kube-api-access-j7t6f\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.001900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.001841    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5002a04-b76d-44cf-8e5f-7063871fe6e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.001900 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:50.001866    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-serving-cert: secret "maas-api-e2e-worker-w1-5f9b95-serving-cert" not found
Oct 30 20:31:50.002072 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:50.001945    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls podName:f5002a04-b76d-44cf-8e5f-7063871fe6e1 nodeName:}" failed. No retries permitted until 2026-10-30 20:31:50.50192392 +0000 UTC m=+736.650974909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls") pod "maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s" (UID: "f5002a04-b76d-44cf-8e5f-7063871fe6e1") : secret "maas-api-e2e-worker-w1-5f9b95-serving-cert" not found
Oct 30 20:31:50.002469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.002444    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5002a04-b76d-44cf-8e5f-7063871fe6e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.004349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.004328    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-metrics-certs\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.032944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.032917    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:50.077867 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.073263    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7t6f\" (UniqueName: \"kubernetes.io/projected/f5002a04-b76d-44cf-8e5f-7063871fe6e1-kube-api-access-j7t6f\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.206648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.206613    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:50.383977 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.383872    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:50.512141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.512109    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.514749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.514724    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:50.553332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.553307    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:50.553866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.553843    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:50.554409 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.554389    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:31:50.716605 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.716581    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 20:31:50.722058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:50.722039    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:51.147753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:51.147730    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"]
Oct 30 20:31:51.149208 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:51.149180    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5002a04_b76d_44cf_8e5f_7063871fe6e1.slice/crio-4a15ac75c4ded9aa1123dd0306c49cd6cc629feb99b351ddbe017cc37640ae17 WatchSource:0}: Error finding container 4a15ac75c4ded9aa1123dd0306c49cd6cc629feb99b351ddbe017cc37640ae17: Status 404 returned error can't find the container with id 4a15ac75c4ded9aa1123dd0306c49cd6cc629feb99b351ddbe017cc37640ae17
Oct 30 20:31:51.791000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:51.790966    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s" event={"ID":"f5002a04-b76d-44cf-8e5f-7063871fe6e1","Type":"ContainerStarted","Data":"4a15ac75c4ded9aa1123dd0306c49cd6cc629feb99b351ddbe017cc37640ae17"}
Oct 30 20:31:52.798030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:52.797997    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s" event={"ID":"f5002a04-b76d-44cf-8e5f-7063871fe6e1","Type":"ContainerStarted","Data":"ec46789e02779f176c14defd1d1fa178fa6f7a14ef082b70d0c82ee9386675d7"}
Oct 30 20:31:52.798429 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:52.798083    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:52.965882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:52.965826    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s" podStartSLOduration=3.018009489 podStartE2EDuration="3.965812268s" totalImagesPullingTime="947.802779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:52.91682913 +0000 UTC m=+739.065880118" watchObservedRunningTime="2026-10-30 20:31:52.965812268 +0000 UTC m=+739.114863256"
Oct 30 20:31:54.896473 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:54.896441    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"]
Oct 30 20:31:54.900691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:54.900667    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:54.954712 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:54.954678    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8e2fc1-serving-cert\""
Oct 30 20:31:54.955416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:54.955386    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8qn\" (UniqueName: \"kubernetes.io/projected/4a899c5a-3146-4b9a-b284-f2959ebe544c-kube-api-access-cp8qn\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:54.955573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:54.955426    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a899c5a-3146-4b9a-b284-f2959ebe544c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:54.955573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:54.955464    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:54.955715 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:54.955574    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.007995 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.007963    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"]
Oct 30 20:31:55.056236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.056207    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8qn\" (UniqueName: \"kubernetes.io/projected/4a899c5a-3146-4b9a-b284-f2959ebe544c-kube-api-access-cp8qn\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.056236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.056241    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a899c5a-3146-4b9a-b284-f2959ebe544c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.056479 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.056260    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.056479 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.056283    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.056479 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:55.056382    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-serving-cert: secret "maas-api-e2e-worker-w4-8e2fc1-serving-cert" not found
Oct 30 20:31:55.056479 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:31:55.056454    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls podName:4a899c5a-3146-4b9a-b284-f2959ebe544c nodeName:}" failed. No retries permitted until 2026-10-30 20:31:55.556431098 +0000 UTC m=+741.705482087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls") pod "maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld" (UID: "4a899c5a-3146-4b9a-b284-f2959ebe544c") : secret "maas-api-e2e-worker-w4-8e2fc1-serving-cert" not found
Oct 30 20:31:55.056954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.056931    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a899c5a-3146-4b9a-b284-f2959ebe544c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.058872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.058853    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.074874 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.074837    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:55.113886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.113861    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8qn\" (UniqueName: \"kubernetes.io/projected/4a899c5a-3146-4b9a-b284-f2959ebe544c-kube-api-access-cp8qn\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.391101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.391061    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:55.560734 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.560700    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.564088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.564061    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.586495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.586472    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:55.795173 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.795144    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:55.814234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.814207    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:55.984307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:55.984272    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:56.194172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:56.194138    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"]
Oct 30 20:31:56.195498 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:31:56.195466    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a899c5a_3146_4b9a_b284_f2959ebe544c.slice/crio-170c2264c63bfd26dde8c475c3be3050f4b1336ce005a0ed47d8897c3a5fcf66 WatchSource:0}: Error finding container 170c2264c63bfd26dde8c475c3be3050f4b1336ce005a0ed47d8897c3a5fcf66: Status 404 returned error can't find the container with id 170c2264c63bfd26dde8c475c3be3050f4b1336ce005a0ed47d8897c3a5fcf66
Oct 30 20:31:56.824550 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:56.824509    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld" event={"ID":"4a899c5a-3146-4b9a-b284-f2959ebe544c","Type":"ContainerStarted","Data":"170c2264c63bfd26dde8c475c3be3050f4b1336ce005a0ed47d8897c3a5fcf66"}
Oct 30 20:31:57.832326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:57.832293    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld" event={"ID":"4a899c5a-3146-4b9a-b284-f2959ebe544c","Type":"ContainerStarted","Data":"8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5"}
Oct 30 20:31:57.832739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:57.832423    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:31:57.950966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:57.950910    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld" podStartSLOduration=3.068374103 podStartE2EDuration="3.950895242s" totalImagesPullingTime="882.521139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:57.948023338 +0000 UTC m=+744.097074326" watchObservedRunningTime="2026-10-30 20:31:57.950895242 +0000 UTC m=+744.099946224"
Oct 30 20:31:58.809547 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:58.809503    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:31:58.918354 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:58.918320    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:59.955886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:31:59.955852    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:32:00.395385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:00.395343    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:03.843944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:03.843915    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 20:32:05.384666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:05.384614    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:07.119063 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:32:07.119024    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:32:08.709964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.709576    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:44258->10.132.0.56:15021: read: connection reset by peer" start-of-body=
Oct 30 20:32:08.709964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.709650    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:44258->10.132.0.56:15021: read: connection reset by peer"
Oct 30 20:32:08.906581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.906493    5002 generic.go:356] "Generic (PLEG): container finished" podID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerID="900d36b82d090acafa474456f7a3c91d019227a1a5a93853dc515666fb097353" exitCode=0
Oct 30 20:32:08.906581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.906568    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" event={"ID":"e6ad1cb3-708e-48f5-abe2-be61ea83e237","Type":"ContainerDied","Data":"900d36b82d090acafa474456f7a3c91d019227a1a5a93853dc515666fb097353"}
Oct 30 20:32:08.906939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.906926    5002 scope.go:122] "RemoveContainer" containerID="900d36b82d090acafa474456f7a3c91d019227a1a5a93853dc515666fb097353"
Oct 30 20:32:08.908227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.908196    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:08.908337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.908282    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:08.908399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:08.908332    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:09.547321 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:09.547278    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:32:09.941005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:09.940905    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" event={"ID":"e6ad1cb3-708e-48f5-abe2-be61ea83e237","Type":"ContainerStarted","Data":"9cc42d1ebdac21aa5df70ebdfc627bccb6b4c5b128aec6beb147740a91ca881f"}
Oct 30 20:32:10.287821 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:10.287783    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:32:10.288050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:10.288030    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:32:10.288108 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:10.288090    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 20:32:10.532739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:10.532702    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:10.724562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:10.724278    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:11.243385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:11.243348    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:11.287781 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:11.287735    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:32:11.287961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:11.287805    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 20:32:11.440566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:11.438996    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:12.287013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:12.286977    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:32:12.287491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:12.287036    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 20:32:13.287308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:13.287262    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:32:13.287763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:13.287338    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 20:32:13.740204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:13.740171    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:14.287837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:14.287799    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:32:14.288293 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:14.287859    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 20:32:14.434582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:14.434546    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:14.643322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:14.643230    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:15.290933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:15.290898    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:32:15.291379 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:15.291312    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:32:15.292218 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:15.292198    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:32:15.835838 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:15.835795    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:20.053421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:20.053382    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:20.586232 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:20.586194    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:20.771593 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:20.771560    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:21.141415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:21.141349    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:24.334076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:24.334034    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:25.905037 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:25.905005    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:28.363310 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:28.363258    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:29.746893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:29.746857    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:32:30.888897 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:30.888805    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:32.135061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:32.135017    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:32.826182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:32.826144    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:33.057939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:33.057904    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:35.605344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:35.605309    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:37.116436 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:32:37.116398    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:32:37.429896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:37.429807    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:38.149698 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:38.149666    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:38.340344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:38.340312    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:39.904085 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:39.904049    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:40.713415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:40.713376    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:42.731697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:42.731661    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:43.511526 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:43.511489    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:43.754469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:43.754435    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:44.094258 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:44.094225    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:32:46.137023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:46.136984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:47.528383 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:47.528313    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:51.002642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:51.002598    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:55.461169 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:55.461119    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:55.713197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:55.713117    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:57.221454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.221417    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"]
Oct 30 20:32:57.225575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.225557    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.330232 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.330204    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"kube-root-ca.crt\""
Oct 30 20:32:57.330617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.330595    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"default-dockercfg-wgmdp\""
Oct 30 20:32:57.330723 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.330691    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"openshift-service-ca.crt\""
Oct 30 20:32:57.330975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.330961    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"premium-w6-4ee72d-kserve-self-signed-certs\""
Oct 30 20:32:57.333132 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.333105    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"]
Oct 30 20:32:57.352314 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.352292    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:57.388550 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.388501    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-dshm\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.388721 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.388628    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-model-cache\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.388721 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.388707    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-home\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.388800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.388732    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tls-certs\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.388858 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.388838    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fv6\" (UniqueName: \"kubernetes.io/projected/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-kube-api-access-f5fv6\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.388922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.388905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tmp-dir\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.489918    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-home\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.489946    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tls-certs\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.489974    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fv6\" (UniqueName: \"kubernetes.io/projected/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-kube-api-access-f5fv6\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.490102    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tmp-dir\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.490172    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-dshm\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.490197    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-model-cache\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490414 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.490383    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-home\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.490456    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tmp-dir\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.490626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.490609    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-model-cache\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.492410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.492390    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-dshm\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.492609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.492592    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tls-certs\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.547537 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.547514    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fv6\" (UniqueName: \"kubernetes.io/projected/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-kube-api-access-f5fv6\") pod \"premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") " pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:57.837848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:57.837755    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:32:58.216929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:58.216902    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"]
Oct 30 20:32:58.217635 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:32:58.217611    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd12f1a1f_6ebd_48c6_9da5_b47e2c8a5ff6.slice/crio-85bf86864908b529dba9cb9c1f910ba4170f5ea547a2dc062429dda73011584a WatchSource:0}: Error finding container 85bf86864908b529dba9cb9c1f910ba4170f5ea547a2dc062429dda73011584a: Status 404 returned error can't find the container with id 85bf86864908b529dba9cb9c1f910ba4170f5ea547a2dc062429dda73011584a
Oct 30 20:32:58.248655 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:58.248627    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" event={"ID":"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6","Type":"ContainerStarted","Data":"85bf86864908b529dba9cb9c1f910ba4170f5ea547a2dc062429dda73011584a"}
Oct 30 20:32:59.255924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:59.255885    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" event={"ID":"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6","Type":"ContainerStarted","Data":"b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c"}
Oct 30 20:32:59.373066 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:59.373016    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podStartSLOduration=2.373002146 podStartE2EDuration="2.373002146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:59.372599826 +0000 UTC m=+805.521650811" watchObservedRunningTime="2026-10-30 20:32:59.373002146 +0000 UTC m=+805.522053133"
Oct 30 20:32:59.838480 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:32:59.838445    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:33:00.503014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:00.502962    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:01.333063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:01.333021    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:02.619828 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:02.619786    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:33:05.297445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:05.297409    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:07.115508 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:33:07.115468    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:33:07.838035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:07.838000    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:33:07.851745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:07.851719    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:33:08.342607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:08.342579    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:33:10.010320 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:10.010279    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:14.812117 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:14.812024    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:17.929973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:17.929905    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:19.506854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:19.506823    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:23.429160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.429123    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"]
Oct 30 20:33:23.433897 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.433879    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.458725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.458691    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v25m\" (UniqueName: \"kubernetes.io/projected/fb61b044-bc78-49d1-9059-46e6919ac07f-kube-api-access-6v25m\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.458884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.458750    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-tmp-dir\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.458884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.458824    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-home\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.458884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.458841    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-model-cache\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.458884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.458855    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb61b044-bc78-49d1-9059-46e6919ac07f-tls-certs\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.458884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.458875    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-dshm\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.488416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.488389    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"]
Oct 30 20:33:23.535670 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.535638    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"openshift-service-ca.crt\""
Oct 30 20:33:23.536061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.536025    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"kube-root-ca.crt\""
Oct 30 20:33:23.536129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.536099    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"simulator-w4-8e2fc1-kserve-self-signed-certs\""
Oct 30 20:33:23.537636 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.537618    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"default-dockercfg-9824w\""
Oct 30 20:33:23.554655 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.554630    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:23.560106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v25m\" (UniqueName: \"kubernetes.io/projected/fb61b044-bc78-49d1-9059-46e6919ac07f-kube-api-access-6v25m\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560199 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560149    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-tmp-dir\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560272 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560258    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-home\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560288    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-model-cache\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560355 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560314    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb61b044-bc78-49d1-9059-46e6919ac07f-tls-certs\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560355 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560347    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-dshm\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560638    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-tmp-dir\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560679    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-model-cache\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.560843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.560828    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-home\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.564177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.564130    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-dshm\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.564287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.564190    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb61b044-bc78-49d1-9059-46e6919ac07f-tls-certs\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.622073 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.622034    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v25m\" (UniqueName: \"kubernetes.io/projected/fb61b044-bc78-49d1-9059-46e6919ac07f-kube-api-access-6v25m\") pod \"simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") " pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:23.748248 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:23.748214    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:24.124984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:24.124950    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"]
Oct 30 20:33:24.125836 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:33:24.125800    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb61b044_bc78_49d1_9059_46e6919ac07f.slice/crio-b0cf944903eb92c2d59c77dda5e40bfe56c16cfca0a4cdc0a767030d65871332 WatchSource:0}: Error finding container b0cf944903eb92c2d59c77dda5e40bfe56c16cfca0a4cdc0a767030d65871332: Status 404 returned error can't find the container with id b0cf944903eb92c2d59c77dda5e40bfe56c16cfca0a4cdc0a767030d65871332
Oct 30 20:33:24.242557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:24.233997    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:24.449664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:24.445994    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:33:24.465563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:24.465210    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" event={"ID":"fb61b044-bc78-49d1-9059-46e6919ac07f","Type":"ContainerStarted","Data":"5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0"}
Oct 30 20:33:24.465563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:24.465260    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" event={"ID":"fb61b044-bc78-49d1-9059-46e6919ac07f","Type":"ContainerStarted","Data":"b0cf944903eb92c2d59c77dda5e40bfe56c16cfca0a4cdc0a767030d65871332"}
Oct 30 20:33:24.585511 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:24.585437    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podStartSLOduration=1.585418201 podStartE2EDuration="1.585418201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:24.582855145 +0000 UTC m=+830.731906132" watchObservedRunningTime="2026-10-30 20:33:24.585418201 +0000 UTC m=+830.734469188"
Oct 30 20:33:24.755592 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:24.748748    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:28.715645 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:28.715612    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:28.900326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:28.900290    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:33.604694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:33.604657    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:33.748456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:33.748427    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:33.761091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:33.761069    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:34.536477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:34.536449    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:33:37.114596 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:33:37.114563    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:33:38.391098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:38.391063    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:43.102896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:43.102736    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:43.796447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:43.794907    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:44.710683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:44.710655    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:33:47.906360 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:47.906305    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:52.591858 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:52.591817    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:53.923311 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:53.923264    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:57.393915 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:33:57.393884    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:02.081869 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:02.081835    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:04.498393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:04.498361    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:34:06.774476 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:06.774437    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:07.114487 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:34:07.114410    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:34:11.161381 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.161350    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"]
Oct 30 20:34:11.165477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.165459    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.253679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.253640    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tmp-dir\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.253847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.253775    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tls-certs\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.253847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.253830    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksp9h\" (UniqueName: \"kubernetes.io/projected/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-kube-api-access-ksp9h\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.253969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.253942    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-dshm\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.254039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.254020    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-model-cache\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.254075 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.254049    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-home\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.270033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.270008    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"kube-root-ca.crt\""
Oct 30 20:34:11.270677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.270659    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"openshift-service-ca.crt\""
Oct 30 20:34:11.270796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.270781    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"default-dockercfg-7xnk5\""
Oct 30 20:34:11.270868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.270818    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"premium-w2-df031b-kserve-self-signed-certs\""
Oct 30 20:34:11.273588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.273568    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"]
Oct 30 20:34:11.291440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.291410    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:11.355460 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355423    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-dshm\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.355673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355471    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-model-cache\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.355673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355491    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-home\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.355673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355585    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tmp-dir\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.355673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355630    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tls-certs\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.355673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355659    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksp9h\" (UniqueName: \"kubernetes.io/projected/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-kube-api-access-ksp9h\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.355981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355960    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-model-cache\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.356026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.355985    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-home\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.356026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.356014    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tmp-dir\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.358004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.357983    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-dshm\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.358382 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.358362    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tls-certs\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.413085 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.413021    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksp9h\" (UniqueName: \"kubernetes.io/projected/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-kube-api-access-ksp9h\") pod \"premium-w2-df031b-kserve-66d54fbd4d-5mpv7\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") " pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.476850 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.476824    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:11.499248 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.499223    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:11.880155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:11.880124    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"]
Oct 30 20:34:11.881688 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:34:11.881662    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b26da6_9ac0_4f6e_a540_cb0dc4a0d019.slice/crio-62e4de510e0daab2ddede6fecabe99f6b225525a401ee7bb2b724a3310d88a26 WatchSource:0}: Error finding container 62e4de510e0daab2ddede6fecabe99f6b225525a401ee7bb2b724a3310d88a26: Status 404 returned error can't find the container with id 62e4de510e0daab2ddede6fecabe99f6b225525a401ee7bb2b724a3310d88a26
Oct 30 20:34:12.757613 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:12.757575    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" event={"ID":"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019","Type":"ContainerStarted","Data":"324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19"}
Oct 30 20:34:12.757613 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:12.757629    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" event={"ID":"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019","Type":"ContainerStarted","Data":"62e4de510e0daab2ddede6fecabe99f6b225525a401ee7bb2b724a3310d88a26"}
Oct 30 20:34:12.877743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:12.877690    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podStartSLOduration=1.877677084 podStartE2EDuration="1.877677084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:12.874315244 +0000 UTC m=+879.023366233" watchObservedRunningTime="2026-10-30 20:34:12.877677084 +0000 UTC m=+879.026728072"
Oct 30 20:34:13.477477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:13.477444    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:16.398446 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:16.398292    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:21.086881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:21.086842    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:21.477449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:21.477421    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:21.490201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:21.490177    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:21.833382 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:21.833307    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:34:23.211996 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:23.211966    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:34:25.742543 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:25.742510    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:29.047453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:29.047396    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:29.238896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:29.238861    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:30.518932 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:30.518892    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:34.093352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:34.093320    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:34:34.094894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:34.094871    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:34:35.301304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:35.301269    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:37.116135 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:34:37.116096    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:34:40.110716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:40.110670    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:41.002320 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.002283    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"]
Oct 30 20:34:41.006326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.006310    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.060800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.060733    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"openshift-service-ca.crt\""
Oct 30 20:34:41.074367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.074334    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwm6x\" (UniqueName: \"kubernetes.io/projected/ae4363cb-f28c-4534-9800-b51e810117fa-kube-api-access-qwm6x\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.074498 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.074390    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-home\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.074562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.074493    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-dshm\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.074636 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.074618    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4363cb-f28c-4534-9800-b51e810117fa-tls-certs\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.074705 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.074689    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-tmp-dir\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.074810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.074785    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-model-cache\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.111079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.111048    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"kube-root-ca.crt\""
Oct 30 20:34:41.111079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.111048    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"premium-w1-5f9b95-kserve-self-signed-certs\""
Oct 30 20:34:41.111488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.111165    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"default-dockercfg-6bjt2\""
Oct 30 20:34:41.114051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.114031    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"]
Oct 30 20:34:41.157240 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.157174    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:41.176262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176230    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-model-cache\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176292    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwm6x\" (UniqueName: \"kubernetes.io/projected/ae4363cb-f28c-4534-9800-b51e810117fa-kube-api-access-qwm6x\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176318    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-home\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176356    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-dshm\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176409    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4363cb-f28c-4534-9800-b51e810117fa-tls-certs\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176449    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-tmp-dir\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176735    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-model-cache\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176749    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-home\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.176898 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.176841    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-tmp-dir\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.178868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.178847    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-dshm\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.178972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.178956    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4363cb-f28c-4534-9800-b51e810117fa-tls-certs\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.233921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.233889    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwm6x\" (UniqueName: \"kubernetes.io/projected/ae4363cb-f28c-4534-9800-b51e810117fa-kube-api-access-qwm6x\") pod \"premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") " pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.319973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.319916    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:41.708855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.708828    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"]
Oct 30 20:34:41.709730 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:34:41.709703    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae4363cb_f28c_4534_9800_b51e810117fa.slice/crio-173584537b0079f30fe6a4c2bbff5cedb3aac86b1c4e3638c420a05489be91e2 WatchSource:0}: Error finding container 173584537b0079f30fe6a4c2bbff5cedb3aac86b1c4e3638c420a05489be91e2: Status 404 returned error can't find the container with id 173584537b0079f30fe6a4c2bbff5cedb3aac86b1c4e3638c420a05489be91e2
Oct 30 20:34:41.952058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.952029    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" event={"ID":"ae4363cb-f28c-4534-9800-b51e810117fa","Type":"ContainerStarted","Data":"42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408"}
Oct 30 20:34:41.952224 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:41.952066    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" event={"ID":"ae4363cb-f28c-4534-9800-b51e810117fa","Type":"ContainerStarted","Data":"173584537b0079f30fe6a4c2bbff5cedb3aac86b1c4e3638c420a05489be91e2"}
Oct 30 20:34:42.071068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:42.071029    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podStartSLOduration=2.071016218 podStartE2EDuration="2.071016218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:42.069156856 +0000 UTC m=+908.218207842" watchObservedRunningTime="2026-10-30 20:34:42.071016218 +0000 UTC m=+908.220067204"
Oct 30 20:34:42.320475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:42.320445    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:44.800507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:44.800469    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:49.506395 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:49.506349    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:51.320716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:51.320676    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:51.338729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:51.338705    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:52.040518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:52.040486    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:34:54.023307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:54.023270    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:34:54.296080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:54.296003    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:55.760419 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:55.760386    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:58.992334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:34:58.992297    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:03.774393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:03.774358    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:07.115628 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:35:07.115590    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:35:08.437881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:08.437815    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:09.111820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:09.111767    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:13.207056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:13.205491    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:17.909930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:17.909871    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:22.606989 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:22.606949    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:22.735875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:22.735840    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:35:27.403205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:27.403165    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:29.400490 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:29.400456    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:32.119161 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:32.119127    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:36.825484 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:36.825454    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:37.120816 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:35:37.120704    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:35:37.949412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:37.949377    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:40.409000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:40.408961    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"]
Oct 30 20:35:40.409410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:40.409174    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy" containerID="cri-o://b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372" gracePeriod=30
Oct 30 20:35:40.462630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:40.462596    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"]
Oct 30 20:35:40.463088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:40.463055    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy" containerID="cri-o://2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f" gracePeriod=30
Oct 30 20:35:41.502455 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:41.502416    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:45.744799 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.744767    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:35:45.864098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864064    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-socket\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864099    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istiod-ca-cert\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864142    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-envoy\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864195    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-credential-socket\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864255    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-data\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864323    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-podinfo\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864322    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-socket" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:45.864620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864393    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-kube-api-access-8lkpz\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-kube-api-access-8lkpz\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864494    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-data" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:45.864620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864555    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-credential-socket" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:45.864620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864563    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-certs\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864569    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istiod-ca-cert" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:45.864620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864597    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-token\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-token\") pod \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\" (UID: \"f1d89af5-52dd-4707-ab01-dd7b28381b9a\") "
Oct 30 20:35:45.864942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.864890    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-certs" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:45.865080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.865035    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.865080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.865062    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.865080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.865077    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.865080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.865092    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.865080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.865105    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.866836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.866807    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-envoy" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:45.867004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.866843    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-kube-api-access-8lkpz" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "kube-api-access-8lkpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:45.867004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.866860    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-podinfo" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:35:45.867004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.866883    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-token" pod "f1d89af5-52dd-4707-ab01-dd7b28381b9a" (UID: "f1d89af5-52dd-4707-ab01-dd7b28381b9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:45.966295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.966226    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.966295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.966251    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkpz\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-kube-api-access-8lkpz\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.966295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.966262    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.966295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:45.966271    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d89af5-52dd-4707-ab01-dd7b28381b9a-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.074751 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.074721    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:35:46.168872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.168837    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-socket\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169062 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.168896    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-credential-socket\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169062 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.168940    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istiod-ca-cert\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169062 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169021    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-podinfo\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169061    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-token\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-token\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169098    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-kube-api-access-6klh4\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-kube-api-access-6klh4\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169175    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-envoy\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169201    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-data\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169196    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-socket" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:46.169249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169213    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-credential-socket" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:46.169522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169302    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-certs\") pod \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\" (UID: \"6bba3496-9380-4fd4-b133-0dd4c4fc48b8\") "
Oct 30 20:35:46.169522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169441    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-data" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:46.169522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169459    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istiod-ca-cert" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:46.169757 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169733    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-certs" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:46.169810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169795    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.169865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169814    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.169865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169830    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.169865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.169844    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.171938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.171916    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-podinfo" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:35:46.171938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.171927    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-envoy" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:46.172115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.172098    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-kube-api-access-6klh4" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "kube-api-access-6klh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:46.172247 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.172226    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-token" pod "6bba3496-9380-4fd4-b133-0dd4c4fc48b8" (UID: "6bba3496-9380-4fd4-b133-0dd4c4fc48b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:46.270939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.270850    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.270939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.270878    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.270939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.270890    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6klh4\" (UniqueName: \"kubernetes.io/projected/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-kube-api-access-6klh4\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.270939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.270904    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.270939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.270915    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba3496-9380-4fd4-b133-0dd4c4fc48b8-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:46.290625 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.290581    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:46.406704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.406669    5002 generic.go:356] "Generic (PLEG): container finished" podID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerID="b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372" exitCode=0
Oct 30 20:35:46.406873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.406725    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"
Oct 30 20:35:46.406873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.406750    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" event={"ID":"6bba3496-9380-4fd4-b133-0dd4c4fc48b8","Type":"ContainerDied","Data":"b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372"}
Oct 30 20:35:46.406873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.406798    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk" event={"ID":"6bba3496-9380-4fd4-b133-0dd4c4fc48b8","Type":"ContainerDied","Data":"e80c1c3f4ad3daf9555e850200008a894f929c083d127bc03bb91c42e671638f"}
Oct 30 20:35:46.406873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.406820    5002 scope.go:122] "RemoveContainer" containerID="b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372"
Oct 30 20:35:46.408693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.408666    5002 generic.go:356] "Generic (PLEG): container finished" podID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerID="2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f" exitCode=0
Oct 30 20:35:46.408822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.408709    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" event={"ID":"f1d89af5-52dd-4707-ab01-dd7b28381b9a","Type":"ContainerDied","Data":"2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f"}
Oct 30 20:35:46.408822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.408722    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"
Oct 30 20:35:46.408822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.408759    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4" event={"ID":"f1d89af5-52dd-4707-ab01-dd7b28381b9a","Type":"ContainerDied","Data":"134ccf037c25784fe0dfc435fc77adba0dec9f94969753a937525fb626a82c5a"}
Oct 30 20:35:46.418609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.418590    5002 scope.go:122] "RemoveContainer" containerID="b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372"
Oct 30 20:35:46.418894 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:35:46.418877    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372\": container with ID starting with b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372 not found: ID does not exist" containerID="b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372"
Oct 30 20:35:46.418961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.418901    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372"} err="failed to get container status \"b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372\": rpc error: code = NotFound desc = could not find container \"b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372\": container with ID starting with b766aeee2f9de1de2527bd46ae549aedbbc41aa87ac27d49f09285d5b0cd3372 not found: ID does not exist"
Oct 30 20:35:46.418961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.418916    5002 scope.go:122] "RemoveContainer" containerID="2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f"
Oct 30 20:35:46.436655 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.436636    5002 scope.go:122] "RemoveContainer" containerID="7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee"
Oct 30 20:35:46.445400 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.445383    5002 scope.go:122] "RemoveContainer" containerID="2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f"
Oct 30 20:35:46.445755 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:35:46.445738    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f\": container with ID starting with 2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f not found: ID does not exist" containerID="2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f"
Oct 30 20:35:46.445861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.445762    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f"} err="failed to get container status \"2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f\": rpc error: code = NotFound desc = could not find container \"2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f\": container with ID starting with 2376b7036b2d2de9f414fb9a9f2ec9124610aafffea2c295c6b4793895d3ad9f not found: ID does not exist"
Oct 30 20:35:46.445861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.445777    5002 scope.go:122] "RemoveContainer" containerID="7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee"
Oct 30 20:35:46.446034 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:35:46.446010    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee\": container with ID starting with 7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee not found: ID does not exist" containerID="7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee"
Oct 30 20:35:46.446095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.446042    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee"} err="failed to get container status \"7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee\": rpc error: code = NotFound desc = could not find container \"7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee\": container with ID starting with 7bea1cdb5858d8b2afbfbdec0b44e450506276c7ed444cf636c933fe60d712ee not found: ID does not exist"
Oct 30 20:35:46.573788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.573723    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"]
Oct 30 20:35:46.586083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.586049    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4"]
Oct 30 20:35:46.691413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.691386    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"]
Oct 30 20:35:46.695049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:46.695028    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk"]
Oct 30 20:35:47.967288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:47.967258    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" path="/var/lib/kubelet/pods/6bba3496-9380-4fd4-b133-0dd4c4fc48b8/volumes"
Oct 30 20:35:47.967761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:47.967746    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" path="/var/lib/kubelet/pods/f1d89af5-52dd-4707-ab01-dd7b28381b9a/volumes"
Oct 30 20:35:51.014903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:51.014873    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:55.718750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:35:55.718715    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:00.363268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:00.363231    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:02.281263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:02.281217    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:36:05.204185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:05.204148    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:07.114915 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:36:07.114881    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:36:09.897436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:09.897391    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:10.660901 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.660867    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"]
Oct 30 20:36:10.661422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.661403    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy"
Oct 30 20:36:10.661523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.661423    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy"
Oct 30 20:36:10.661523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.661473    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy"
Oct 30 20:36:10.661523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.661481    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy"
Oct 30 20:36:10.661707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.661627    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba3496-9380-4fd4-b133-0dd4c4fc48b8" containerName="istio-proxy"
Oct 30 20:36:10.661707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.661642    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy"
Oct 30 20:36:10.661707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.661654    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy"
Oct 30 20:36:10.666446 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.666426    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.711612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.711586    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"]
Oct 30 20:36:10.712063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.712050    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy"
Oct 30 20:36:10.712124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.712064    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d89af5-52dd-4707-ab01-dd7b28381b9a" containerName="istio-proxy"
Oct 30 20:36:10.715420 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.715401    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.729088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.729064    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7k2d5\""
Oct 30 20:36:10.770622 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.770593    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"]
Oct 30 20:36:10.775741 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.775717    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"]
Oct 30 20:36:10.817575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.817526    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.817747 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.817728    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.817839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.817768    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.817839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.817801    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.817952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.817883    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.817952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.817920    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.818034 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.817990    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.818034 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818018    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.818102 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818067    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037d4d37-70b9-4335-9b96-4e5c740a48bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.818144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818098    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.818144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818133    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhc8\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-kube-api-access-rrhc8\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.818208 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818171    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.818240 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818226    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.818285 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818254    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.818362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818342    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tfr\" (UniqueName: \"kubernetes.io/projected/037d4d37-70b9-4335-9b96-4e5c740a48bc-kube-api-access-22tfr\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.818427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818393    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.818585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818444    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.818585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.818468    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.919666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919548    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.919666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919592    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.919666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919625    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.919666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919655    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tfr\" (UniqueName: \"kubernetes.io/projected/037d4d37-70b9-4335-9b96-4e5c740a48bc-kube-api-access-22tfr\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919693    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919728    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919754    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919787    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919811    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919842    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919870    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919899    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919961    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.919984    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920017    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920060    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037d4d37-70b9-4335-9b96-4e5c740a48bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920097    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920146    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhc8\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-kube-api-access-rrhc8\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920289    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.920336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920311    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.921142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920451    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.921142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920504    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.921142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.920908    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.921743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.921186    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037d4d37-70b9-4335-9b96-4e5c740a48bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.921743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.921300    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.921743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.921547    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.922111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.922088    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.922719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.922693    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.923695 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.923668    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.923803 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.923705    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.924427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.924407    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.991849 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.991806    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037d4d37-70b9-4335-9b96-4e5c740a48bc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.992019 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.991996    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:10.992848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.992818    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tfr\" (UniqueName: \"kubernetes.io/projected/037d4d37-70b9-4335-9b96-4e5c740a48bc-kube-api-access-22tfr\") pod \"maas-default-gateway-openshift-default-6886b96f9-mr976\" (UID: \"037d4d37-70b9-4335-9b96-4e5c740a48bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:10.993739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:10.993723    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhc8\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-kube-api-access-rrhc8\") pod \"maas-default-gateway-openshift-default-6886b96f9-hjtwx\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:11.029970 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.029951    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:11.279890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.279862    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:11.458486 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.458456    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"]
Oct 30 20:36:11.458772 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:36:11.458734    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fae8f0e_17a6_4689_8da3_7702b3e4842e.slice/crio-362131bbbc9bf83ff28c1cdf9e75c7cb50a887fc6bd40f7ed24349ca9a51e0d7 WatchSource:0}: Error finding container 362131bbbc9bf83ff28c1cdf9e75c7cb50a887fc6bd40f7ed24349ca9a51e0d7: Status 404 returned error can't find the container with id 362131bbbc9bf83ff28c1cdf9e75c7cb50a887fc6bd40f7ed24349ca9a51e0d7
Oct 30 20:36:11.461062 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.461027    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:11.461155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.461096    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:11.461155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.461124    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:11.491588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.491560    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:11.565115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.565081    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" event={"ID":"6fae8f0e-17a6-4689-8da3-7702b3e4842e","Type":"ContainerStarted","Data":"e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a"}
Oct 30 20:36:11.565296 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.565157    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" event={"ID":"6fae8f0e-17a6-4689-8da3-7702b3e4842e","Type":"ContainerStarted","Data":"362131bbbc9bf83ff28c1cdf9e75c7cb50a887fc6bd40f7ed24349ca9a51e0d7"}
Oct 30 20:36:11.657759 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.657723    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"]
Oct 30 20:36:11.660383 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:36:11.660355    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037d4d37_70b9_4335_9b96_4e5c740a48bc.slice/crio-8575c67666b3ddfa4eb10f28f7bd64448a01c9aaa1cb0d9673dcbea321676027 WatchSource:0}: Error finding container 8575c67666b3ddfa4eb10f28f7bd64448a01c9aaa1cb0d9673dcbea321676027: Status 404 returned error can't find the container with id 8575c67666b3ddfa4eb10f28f7bd64448a01c9aaa1cb0d9673dcbea321676027
Oct 30 20:36:11.662165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.662135    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:11.662244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.662199    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:11.662244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.662230    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:11.688012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:11.687971    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podStartSLOduration=1.68795788 podStartE2EDuration="1.68795788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:11.685753636 +0000 UTC m=+997.834804623" watchObservedRunningTime="2026-10-30 20:36:11.68795788 +0000 UTC m=+997.837008867"
Oct 30 20:36:12.033875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:12.033843    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:12.037286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:12.037252    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:12.037453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:12.037312    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:12.581005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:12.580963    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" event={"ID":"037d4d37-70b9-4335-9b96-4e5c740a48bc","Type":"ContainerStarted","Data":"c78312d9437ae91391e335c29fa905c827381fd967c5009eab19d03675abe300"}
Oct 30 20:36:12.581207 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:12.581020    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" event={"ID":"037d4d37-70b9-4335-9b96-4e5c740a48bc","Type":"ContainerStarted","Data":"8575c67666b3ddfa4eb10f28f7bd64448a01c9aaa1cb0d9673dcbea321676027"}
Oct 30 20:36:13.031946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:13.031906    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:13.032211 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:13.032181    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:13.280523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:13.280493    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:13.282028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:13.281962    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:36:13.282028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:13.282011    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:36:14.032917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:14.032882    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:14.033209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:14.033180    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:14.281349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:14.281313    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:36:14.281768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:14.281366    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:36:14.822594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:14.809783    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:15.030837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:15.030802    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:15.031115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:15.031086    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:15.280890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:15.280853    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:36:15.281096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:15.280908    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:36:15.858558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:15.857520    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:16.032567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:16.031839    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:16.032567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:16.031902    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:16.065313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:16.065231    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:16.280399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:16.280362    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:36:16.280605 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:16.280429    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:36:17.033556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:17.032832    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:17.033556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:17.032888    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:17.280574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:17.280517    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:36:17.280813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:17.280784    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:36:18.032155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:18.032127    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:18.032418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:18.032393    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:18.281249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:18.281214    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:36:18.281760 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:18.281270    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:36:19.030914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:19.030425    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:36:19.030914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:19.030485    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:36:19.280692 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:19.280649    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:36:19.280836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:19.280728    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:36:19.635572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:19.631892    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:20.748831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:20.748796    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:20.750024 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:20.750002    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:20.752886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:20.752864    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:36:20.756336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:20.756258    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:20.756966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:20.756950    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:20.762284 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:20.762126    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976"
Oct 30 20:36:20.922173 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:20.922127    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podStartSLOduration=10.922114671 podStartE2EDuration="10.922114671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:12.712851938 +0000 UTC m=+998.861902925" watchObservedRunningTime="2026-10-30 20:36:20.922114671 +0000 UTC m=+1007.071165657"
Oct 30 20:36:22.784296 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.784271    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"]
Oct 30 20:36:22.788254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.788237    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818019 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.817991    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818033    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818082    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9823d53-c5da-406a-a02c-1509bf0d2edc-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818147    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8cw\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-kube-api-access-7x8cw\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818320 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818229    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818468    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818594    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818652    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.818726 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.818701    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.841524 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.841497    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a78bf4-openshift-default-dockercfg-cn9sv\""
Oct 30 20:36:22.894961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.894934    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"]
Oct 30 20:36:22.920450 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920422    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920515    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920572    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920610    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920655    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920688    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920715    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9823d53-c5da-406a-a02c-1509bf0d2edc-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920763    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8cw\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-kube-api-access-7x8cw\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.920992 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920805    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.921045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.920999    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.921096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.921040    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.921219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.921197    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.921467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.921445    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.921708 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.921676    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9823d53-c5da-406a-a02c-1509bf0d2edc-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.923402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.923375    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.924646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.924627    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.978267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.978245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:22.978432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:22.978414    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8cw\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-kube-api-access-7x8cw\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:23.101911 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.101858    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:23.530518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.530490    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"]
Oct 30 20:36:23.532682 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:36:23.532657    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9823d53_c5da_406a_a02c_1509bf0d2edc.slice/crio-4d0940bdbf3e972b2dc719c2337d6cf15d1c2d4857371feb18aa384cd71f3e03 WatchSource:0}: Error finding container 4d0940bdbf3e972b2dc719c2337d6cf15d1c2d4857371feb18aa384cd71f3e03: Status 404 returned error can't find the container with id 4d0940bdbf3e972b2dc719c2337d6cf15d1c2d4857371feb18aa384cd71f3e03
Oct 30 20:36:23.535671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.535633    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:23.535766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.535716    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:23.535766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.535755    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:23.734451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.734410    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" event={"ID":"f9823d53-c5da-406a-a02c-1509bf0d2edc","Type":"ContainerStarted","Data":"24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a"}
Oct 30 20:36:23.734657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.734507    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" event={"ID":"f9823d53-c5da-406a-a02c-1509bf0d2edc","Type":"ContainerStarted","Data":"4d0940bdbf3e972b2dc719c2337d6cf15d1c2d4857371feb18aa384cd71f3e03"}
Oct 30 20:36:23.856873 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:23.856767    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podStartSLOduration=1.8567490229999999 podStartE2EDuration="1.856749023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:23.852822859 +0000 UTC m=+1010.001873848" watchObservedRunningTime="2026-10-30 20:36:23.856749023 +0000 UTC m=+1010.005800011"
Oct 30 20:36:24.105980 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:24.105932    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:24.110637 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:24.110428    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:36:24.110637 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:24.110496    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:36:24.294046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:24.294011    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:25.102345 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:25.102313    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:36:25.102769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:25.102358    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:36:26.102816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:26.102788    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:36:26.103192 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:26.102839    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:36:27.103048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:27.102963    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:36:27.103048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:27.103027    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:36:28.102831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:28.102798    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:36:28.102998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:28.102851    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:36:29.102294 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:29.102262    5002 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:36:29.102687 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:29.102309    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:36:29.147133 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:29.147091    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:29.728697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:29.728663    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:30.106905 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:30.106832    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:30.107292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:30.107276    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:30.107998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:30.107977    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:36:33.931348 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:33.931295    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:37.114702 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:36:37.114645    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:36:37.817420 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.817392    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"]
Oct 30 20:36:37.822668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.822648    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888366    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888521 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888406    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888563    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888587    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888616    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888687 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888663    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888734 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888722    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888746    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscmx\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-kube-api-access-bscmx\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.888802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.888766    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.913902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.913862    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"]
Oct 30 20:36:37.989689 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.989661    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.989870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.989705    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.989870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.989750    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.989870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.989781    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.989870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.989808    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.989936    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscmx\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-kube-api-access-bscmx\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.989990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.990058    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.990091    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.990098    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.990135    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990354 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.990273    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990401 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.990381    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.990575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.990551    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.992581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.992523    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:37.992733 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:37.992718    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:38.048143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.048122    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscmx\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-kube-api-access-bscmx\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:38.048352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.048332    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:38.137372 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.137310    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:38.581192 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.581163    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"]
Oct 30 20:36:38.581786 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:36:38.581762    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice/crio-a61b44dbed661ad7b1b642d74eb2b18a5d0a049d92738a6450cfa8ce98bbc18a WatchSource:0}: Error finding container a61b44dbed661ad7b1b642d74eb2b18a5d0a049d92738a6450cfa8ce98bbc18a: Status 404 returned error can't find the container with id a61b44dbed661ad7b1b642d74eb2b18a5d0a049d92738a6450cfa8ce98bbc18a
Oct 30 20:36:38.584077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.584044    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:38.584153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.584113    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:38.584153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.584144    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:38.676859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.676822    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:38.838287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.838207    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" event={"ID":"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487","Type":"ContainerStarted","Data":"134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7"}
Oct 30 20:36:38.838460 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.838301    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" event={"ID":"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487","Type":"ContainerStarted","Data":"a61b44dbed661ad7b1b642d74eb2b18a5d0a049d92738a6450cfa8ce98bbc18a"}
Oct 30 20:36:38.962215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:38.962159    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podStartSLOduration=1.962145311 podStartE2EDuration="1.962145311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:38.959698013 +0000 UTC m=+1025.108749000" watchObservedRunningTime="2026-10-30 20:36:38.962145311 +0000 UTC m=+1025.111196298"
Oct 30 20:36:39.138013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:39.137926    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:39.139723 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:39.139689    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:36:39.139861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:39.139749    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 20:36:40.137871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:40.137838    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:36:40.138096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:40.137889    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 20:36:40.159247 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:40.159213    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:36:41.137945 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:41.137906    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:36:41.138363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:41.138330    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 20:36:42.137673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:42.137642    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:36:42.137842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:42.137689    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 20:36:42.576650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:42.576620    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:42.762775 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:42.762745    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:43.138122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:43.138082    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:36:43.138343 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:43.138150    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 20:36:43.442413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:43.442331    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:44.137909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:44.137869    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:36:44.138520 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:44.138479    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 20:36:45.138347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:45.138311    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:36:45.138763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:45.138371    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 20:36:46.166273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:46.166245    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:46.166780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:46.166763    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:46.167466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:46.167448    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:36:48.284562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:48.284518    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:53.000727 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:53.000689    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:57.717033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:36:57.716995    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:00.689283 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.689253    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"]
Oct 30 20:37:00.693457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.693439    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.757979 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.757950    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-model-cache\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.758175 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.757995    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tmp-dir\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.758175 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.758144    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jcq\" (UniqueName: \"kubernetes.io/projected/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-kube-api-access-46jcq\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.758299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.758183    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tls-certs\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.758299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.758262    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-home\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.758398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.758380    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-dshm\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.791670 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.791643    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"]
Oct 30 20:37:00.798959 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.798936    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"openshift-service-ca.crt\""
Oct 30 20:37:00.798959 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.798935    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"kube-root-ca.crt\""
Oct 30 20:37:00.799105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.799011    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"distinct-w5-a78bf4-kserve-self-signed-certs\""
Oct 30 20:37:00.799140 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.799121    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"default-dockercfg-999rs\""
Oct 30 20:37:00.838100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.838075    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:00.875935 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.875903    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jcq\" (UniqueName: \"kubernetes.io/projected/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-kube-api-access-46jcq\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.876136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.875941    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tls-certs\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.876136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.875981    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-home\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.876136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.876048    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-dshm\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.876136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.876129    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-model-cache\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.876369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.876148    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tmp-dir\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.876664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.876635    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-model-cache\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.876794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.876743    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-home\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.877006 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.876976    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tmp-dir\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.878918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.878897    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-dshm\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.879299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.879280    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tls-certs\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:00.936560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:00.936513    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jcq\" (UniqueName: \"kubernetes.io/projected/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-kube-api-access-46jcq\") pod \"distinct-w5-a78bf4-kserve-7d47585f44-tjbqz\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:01.006492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:01.006467    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:01.384378 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:01.384342    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"]
Oct 30 20:37:01.384786 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:37:01.384759    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26bd6dcf_4485_4d9c_8bd3_02e8f1f0af4a.slice/crio-e148639e328f9c1e8e01b2eab8bce179fef2ae07c0c9e4d9fcadaff68413314f WatchSource:0}: Error finding container e148639e328f9c1e8e01b2eab8bce179fef2ae07c0c9e4d9fcadaff68413314f: Status 404 returned error can't find the container with id e148639e328f9c1e8e01b2eab8bce179fef2ae07c0c9e4d9fcadaff68413314f
Oct 30 20:37:01.996762 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:01.996719    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" event={"ID":"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a","Type":"ContainerStarted","Data":"9e8150813c2419f6712694f013b0c07cdaf2b38142ce101a338441d4b00e7689"}
Oct 30 20:37:01.997165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:01.996771    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" event={"ID":"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a","Type":"ContainerStarted","Data":"e148639e328f9c1e8e01b2eab8bce179fef2ae07c0c9e4d9fcadaff68413314f"}
Oct 30 20:37:02.007035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:02.007007    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:02.113968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:02.113924    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podStartSLOduration=2.113910013 podStartE2EDuration="2.113910013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:02.113880338 +0000 UTC m=+1048.262931325" watchObservedRunningTime="2026-10-30 20:37:02.113910013 +0000 UTC m=+1048.262960998"
Oct 30 20:37:02.496960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:02.496906    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:06.764862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:06.764832    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:07.116443 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:37:07.116349    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:37:07.219474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:07.219386    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.202667 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:10.202631    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:37:11.007604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:11.007567    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:11.026986 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:11.026956    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:11.079700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:11.079670    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:37:12.208632 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:12.208594    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.352275 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.352238    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"]
Oct 30 20:37:16.356502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.356480    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.436905 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.436877    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tls-certs\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.437071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.436922    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-dshm\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.437141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.437093    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnfs\" (UniqueName: \"kubernetes.io/projected/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-kube-api-access-dbnfs\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.437204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.437150    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-home\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.437204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.437186    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tmp-dir\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.437302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.437254    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-model-cache\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.461459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.461435    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"distinct-w2-df031b-kserve-self-signed-certs\""
Oct 30 20:37:16.464350 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.464332    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"]
Oct 30 20:37:16.496421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.496399    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.538131 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538102    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tmp-dir\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538301 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538148    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-model-cache\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538301 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538215    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tls-certs\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538301 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538251    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-dshm\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538314    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnfs\" (UniqueName: \"kubernetes.io/projected/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-kube-api-access-dbnfs\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538358    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-home\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538557    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tmp-dir\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538667 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538643    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-model-cache\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.538751 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.538736    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-home\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.540681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.540657    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-dshm\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.540870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.540851    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tls-certs\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.596768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.596744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnfs\" (UniqueName: \"kubernetes.io/projected/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-kube-api-access-dbnfs\") pod \"distinct-w2-df031b-kserve-5bf4598479-8m9v5\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.668495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.668434    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:16.915391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:16.915358    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:17.051023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:17.050997    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"]
Oct 30 20:37:17.051260 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:37:17.051240    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79a2ec9a_b6e5_42fc_ad00_3cd0267db026.slice/crio-1aa5a0532a570fe7410a229c044960333deb4c894edaa27c6e41bdf9c4433a29 WatchSource:0}: Error finding container 1aa5a0532a570fe7410a229c044960333deb4c894edaa27c6e41bdf9c4433a29: Status 404 returned error can't find the container with id 1aa5a0532a570fe7410a229c044960333deb4c894edaa27c6e41bdf9c4433a29
Oct 30 20:37:17.106724 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:17.106701    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" event={"ID":"79a2ec9a-b6e5-42fc-ad00-3cd0267db026","Type":"ContainerStarted","Data":"1aa5a0532a570fe7410a229c044960333deb4c894edaa27c6e41bdf9c4433a29"}
Oct 30 20:37:18.114987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:18.114945    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" event={"ID":"79a2ec9a-b6e5-42fc-ad00-3cd0267db026","Type":"ContainerStarted","Data":"e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054"}
Oct 30 20:37:18.234809 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:18.234759    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podStartSLOduration=2.234744786 podStartE2EDuration="2.234744786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:18.232713945 +0000 UTC m=+1064.381764932" watchObservedRunningTime="2026-10-30 20:37:18.234744786 +0000 UTC m=+1064.383795772"
Oct 30 20:37:18.669214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:18.669188    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:21.785090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:21.785057    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:26.433778 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:26.433707    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:26.671094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:26.671066    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:26.686093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:26.686024    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:27.186125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:27.186092    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:37:28.051889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:28.051852    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:30.066913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:30.066877    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:31.236676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:31.236642    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:35.000792 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:35.000758    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:35.993141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:35.993108    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.114791 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:37:37.114761    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:37:40.686095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:40.686062    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:45.390007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:45.389966    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:46.011933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:46.011903    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:37:50.102250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:50.102189    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:54.892523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:54.892482    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:58.667663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.667632    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"]
Oct 30 20:37:58.671726 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.671710    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.761253 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.761220    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"]
Oct 30 20:37:58.776483 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.776455    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"distinct-w6-4ee72d-kserve-self-signed-certs\""
Oct 30 20:37:58.808705 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.808682    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:58.852566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.850054    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b34596f-6026-447b-88a8-8e61ebefaede-tls-certs\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.852566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.850307    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vm9m\" (UniqueName: \"kubernetes.io/projected/9b34596f-6026-447b-88a8-8e61ebefaede-kube-api-access-2vm9m\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.852566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.850527    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-tmp-dir\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.852566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.850767    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-home\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.852566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.850953    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-model-cache\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.852566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.851220    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-dshm\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954194    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-dshm\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954247    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b34596f-6026-447b-88a8-8e61ebefaede-tls-certs\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954302    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vm9m\" (UniqueName: \"kubernetes.io/projected/9b34596f-6026-447b-88a8-8e61ebefaede-kube-api-access-2vm9m\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954355    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-tmp-dir\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954384    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-home\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954398    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-model-cache\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954781    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-model-cache\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.954802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.954795    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-home\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.955068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.955050    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-tmp-dir\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.957143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.957117    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-dshm\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:58.957347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:58.957328    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b34596f-6026-447b-88a8-8e61ebefaede-tls-certs\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:59.011786 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:59.011754    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vm9m\" (UniqueName: \"kubernetes.io/projected/9b34596f-6026-447b-88a8-8e61ebefaede-kube-api-access-2vm9m\") pod \"distinct-w6-4ee72d-kserve-696686446b-k46ss\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:59.283955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:59.283919    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:37:59.592881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:59.592808    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:59.659753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:37:59.659721    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"]
Oct 30 20:37:59.659880 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:37:59.659803    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b34596f_6026_447b_88a8_8e61ebefaede.slice/crio-db955a1be9392d7a213ae9599ad5713d50b6aff03bf62c7ad36fd266126fd354 WatchSource:0}: Error finding container db955a1be9392d7a213ae9599ad5713d50b6aff03bf62c7ad36fd266126fd354: Status 404 returned error can't find the container with id db955a1be9392d7a213ae9599ad5713d50b6aff03bf62c7ad36fd266126fd354
Oct 30 20:38:00.392721 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:00.392691    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" event={"ID":"9b34596f-6026-447b-88a8-8e61ebefaede","Type":"ContainerStarted","Data":"6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2"}
Oct 30 20:38:00.393112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:00.392730    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" event={"ID":"9b34596f-6026-447b-88a8-8e61ebefaede","Type":"ContainerStarted","Data":"db955a1be9392d7a213ae9599ad5713d50b6aff03bf62c7ad36fd266126fd354"}
Oct 30 20:38:00.509586 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:00.509506    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podStartSLOduration=2.509493376 podStartE2EDuration="2.509493376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:00.508500867 +0000 UTC m=+1106.657551855" watchObservedRunningTime="2026-10-30 20:38:00.509493376 +0000 UTC m=+1106.658544365"
Oct 30 20:38:01.284306 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:01.284276    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:38:04.377069 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:04.377034    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:07.115629 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:38:07.115584    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:38:09.091481 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:09.091443    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:09.284104 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:09.284069    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:38:09.296144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:09.296124    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:38:09.459560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:09.459506    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:38:13.419771 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:13.419694    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:38:13.927096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:13.927056    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:16.795971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:16.795898    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:18.797993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:18.797960    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:23.597172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:23.597121    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:28.433193 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:28.433148    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:30.398081 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:30.398044    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:33.100396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:33.100361    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:37.114778 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:38:37.114745    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:38:37.810172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:37.810121    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:39.240234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.240199    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"]
Oct 30 20:38:39.244307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.244282    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.345856 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.345824    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"distinct-2-w5-a78bf4-kserve-self-signed-certs\""
Oct 30 20:38:39.351734 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.351703    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"]
Oct 30 20:38:39.369312 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.369282    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5d46\" (UniqueName: \"kubernetes.io/projected/f916d575-cbfd-4e99-9b00-e6d70dfdd872-kube-api-access-p5d46\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.369443 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.369327    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-model-cache\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.369519 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.369492    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tls-certs\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.369604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.369579    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-home\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.369652 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.369618    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-dshm\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.369773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.369756    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tmp-dir\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.382334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.382175    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:39.470854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.470822    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tmp-dir\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.470917    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5d46\" (UniqueName: \"kubernetes.io/projected/f916d575-cbfd-4e99-9b00-e6d70dfdd872-kube-api-access-p5d46\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.470943    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-model-cache\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.471085    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tls-certs\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.471133    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-home\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.471174    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-dshm\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471345 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.471274    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tmp-dir\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471461 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.471440    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-model-cache\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.471517 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.471455    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-home\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.473688 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.473660    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-dshm\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.473942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.473921    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tls-certs\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.529236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.529184    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5d46\" (UniqueName: \"kubernetes.io/projected/f916d575-cbfd-4e99-9b00-e6d70dfdd872-kube-api-access-p5d46\") pod \"distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") " pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.556216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.556193    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:39.986612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:39.986578    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"]
Oct 30 20:38:39.987954 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:38:39.987924    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf916d575_cbfd_4e99_9b00_e6d70dfdd872.slice/crio-9a9275343f433e148c8169c21658e05a321897e28acbf66b398640e2270687a3 WatchSource:0}: Error finding container 9a9275343f433e148c8169c21658e05a321897e28acbf66b398640e2270687a3: Status 404 returned error can't find the container with id 9a9275343f433e148c8169c21658e05a321897e28acbf66b398640e2270687a3
Oct 30 20:38:40.658026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:40.657986    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" event={"ID":"f916d575-cbfd-4e99-9b00-e6d70dfdd872","Type":"ContainerStarted","Data":"b1864d37532fddd56be5d74fc98e867e9d58a0434c1f7aec28642c8e5a3bbdb0"}
Oct 30 20:38:40.658026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:40.658029    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" event={"ID":"f916d575-cbfd-4e99-9b00-e6d70dfdd872","Type":"ContainerStarted","Data":"9a9275343f433e148c8169c21658e05a321897e28acbf66b398640e2270687a3"}
Oct 30 20:38:40.780580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:40.780517    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" podStartSLOduration=1.7805061599999998 podStartE2EDuration="1.78050616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:40.778815242 +0000 UTC m=+1146.927866228" watchObservedRunningTime="2026-10-30 20:38:40.78050616 +0000 UTC m=+1146.929557145"
Oct 30 20:38:41.557147 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:41.557117    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:42.720391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:42.711559    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:47.402940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:47.402899    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:49.377942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:49.377905    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:38:49.556562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:49.556511    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:49.574181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:49.574158    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:49.724273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:49.724250    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:38:52.192052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.192017    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:52.468783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.468754    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"]
Oct 30 20:38:52.472906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.472887    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.506830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.506805    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7v45\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-kube-api-access-n7v45\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.506932 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.506845    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.506932 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.506902    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.506932 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.506920    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.507034 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.506964    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.507112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.507094    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.507147 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.507121    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.507242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.507224    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.507289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.507258    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b08158-f394-4fed-84cf-9a3e0a476d3e-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.525586 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.525566    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0a78ac-openshift-default-dockercfg-6pwsb\""
Oct 30 20:38:52.580470 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.580443    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"]
Oct 30 20:38:52.608922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.608889    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609059 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.608933    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609059 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.608993    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609059 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609033    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b08158-f394-4fed-84cf-9a3e0a476d3e-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609071    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7v45\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-kube-api-access-n7v45\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609117    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609170    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609208    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609290    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609340    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.609741 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.609721    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.610050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.610025    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.610135 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.610054    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b08158-f394-4fed-84cf-9a3e0a476d3e-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.610200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.610164    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.612452 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.612426    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.612885 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.612866    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.666924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.666899    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.667213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.667193    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7v45\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-kube-api-access-n7v45\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:52.786928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:52.786858    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:53.219007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.218980    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"]
Oct 30 20:38:53.220814 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:38:53.220784    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b08158_f394_4fed_84cf_9a3e0a476d3e.slice/crio-b7fd9125eb6c553d430eeedf7272f8eadf1648180f17c78bdb01bdcf990357bb WatchSource:0}: Error finding container b7fd9125eb6c553d430eeedf7272f8eadf1648180f17c78bdb01bdcf990357bb: Status 404 returned error can't find the container with id b7fd9125eb6c553d430eeedf7272f8eadf1648180f17c78bdb01bdcf990357bb
Oct 30 20:38:53.223094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.223054    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:53.223220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.223140    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:53.223220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.223174    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:53.748229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.748191    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" event={"ID":"39b08158-f394-4fed-84cf-9a3e0a476d3e","Type":"ContainerStarted","Data":"6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54"}
Oct 30 20:38:53.748439 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.748260    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" event={"ID":"39b08158-f394-4fed-84cf-9a3e0a476d3e","Type":"ContainerStarted","Data":"b7fd9125eb6c553d430eeedf7272f8eadf1648180f17c78bdb01bdcf990357bb"}
Oct 30 20:38:53.787815 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.787789    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:38:53.870255 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:53.870209    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podStartSLOduration=1.870195831 podStartE2EDuration="1.870195831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:53.867825713 +0000 UTC m=+1160.016876702" watchObservedRunningTime="2026-10-30 20:38:53.870195831 +0000 UTC m=+1160.019246817"
Oct 30 20:38:54.788699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:54.788666    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:38:54.789174 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:54.788726    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 20:38:55.787738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:55.787700    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:38:55.787913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:55.787757    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 20:38:56.788009 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:56.787975    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:38:56.788379 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:56.788032    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 20:38:56.931560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:56.923386    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:57.787462 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:57.787432    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:38:57.787676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:57.787508    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 20:38:58.787415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:58.787382    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:38:58.787841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:58.787443    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 20:38:59.787414 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:59.787370    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:38:59.787645 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:38:59.787445    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 20:39:00.791973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:00.791945    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:39:00.792656 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:00.792629    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:39:00.793289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:00.793270    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:39:01.588021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:01.587990    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:06.305685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:06.305650    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:07.115586 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:39:07.115550    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:39:10.975205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:10.975152    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:15.705934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:15.705899    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:19.681417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:19.681381    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:20.598005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:20.597975    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:25.293020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:25.292984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:29.944517 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:29.944485    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:30.760228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:30.760200    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:33.992505 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:33.992476    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:34.137485 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:34.137456    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:39:34.142982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:34.142961    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:39:34.813325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:34.813288    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:34.991796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:34.991767    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:37.123762 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:39:37.123725    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:39:39.573008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:39.572964    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:41.992671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:41.992638    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:42.224790 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:42.224757    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:39:44.316561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:44.315251    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:49.193841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:49.193813    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:51.874747 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:51.874712    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"]
Oct 30 20:39:51.878957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:51.878934    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:51.983855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:51.983830    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"distinct-2-w1-5f9b95-kserve-self-signed-certs\""
Oct 30 20:39:51.987009 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:51.986990    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"]
Oct 30 20:39:52.002769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.002742    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:52.065765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.065729    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-model-cache\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.065974 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.065820    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7xv\" (UniqueName: \"kubernetes.io/projected/231ea425-b247-4ce1-a74f-18354c227a8f-kube-api-access-jj7xv\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.065974 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.065905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231ea425-b247-4ce1-a74f-18354c227a8f-tls-certs\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.066083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.066054    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-home\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.066139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.066085    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-tmp-dir\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.066139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.066123    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-dshm\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.167937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.167868    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-model-cache\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.167937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.167898    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7xv\" (UniqueName: \"kubernetes.io/projected/231ea425-b247-4ce1-a74f-18354c227a8f-kube-api-access-jj7xv\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.167937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.167931    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231ea425-b247-4ce1-a74f-18354c227a8f-tls-certs\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.168199 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.167965    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-home\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.168199 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.167979    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-tmp-dir\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.168199 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.167998    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-dshm\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.168363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.168272    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-model-cache\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.168363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.168333    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-home\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.168437 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.168383    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-tmp-dir\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.170396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.170374    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-dshm\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.170554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.170526    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231ea425-b247-4ce1-a74f-18354c227a8f-tls-certs\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.227282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.227251    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7xv\" (UniqueName: \"kubernetes.io/projected/231ea425-b247-4ce1-a74f-18354c227a8f-kube-api-access-jj7xv\") pod \"distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.496427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.496399    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:52.882429 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:52.882390    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"]
Oct 30 20:39:52.882874 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:39:52.882486    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod231ea425_b247_4ce1_a74f_18354c227a8f.slice/crio-c0acb2de4344d32e4770aa2b8834845283fc24120dcd20ebe1a06797c8cd04a6 WatchSource:0}: Error finding container c0acb2de4344d32e4770aa2b8834845283fc24120dcd20ebe1a06797c8cd04a6: Status 404 returned error can't find the container with id c0acb2de4344d32e4770aa2b8834845283fc24120dcd20ebe1a06797c8cd04a6
Oct 30 20:39:53.175067 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:53.174973    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" event={"ID":"231ea425-b247-4ce1-a74f-18354c227a8f","Type":"ContainerStarted","Data":"dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050"}
Oct 30 20:39:53.175067 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:53.175018    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" event={"ID":"231ea425-b247-4ce1-a74f-18354c227a8f","Type":"ContainerStarted","Data":"c0acb2de4344d32e4770aa2b8834845283fc24120dcd20ebe1a06797c8cd04a6"}
Oct 30 20:39:53.316113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:53.316053    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podStartSLOduration=2.316035942 podStartE2EDuration="2.316035942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:53.314255066 +0000 UTC m=+1219.463306054" watchObservedRunningTime="2026-10-30 20:39:53.316035942 +0000 UTC m=+1219.465086929"
Oct 30 20:39:53.496510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:53.496479    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:39:53.848605 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:53.848513    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:58.442856 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:58.442822    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:58.622912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:58.622881    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:58.819563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:39:58.819503    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:02.496836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:02.496806    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:40:02.508846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:02.508822    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:40:03.252884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:03.252861    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:40:03.408444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:03.408415    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:07.115732 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:40:07.115693    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:40:08.160863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:08.160830    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:09.439806 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.439769    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"]
Oct 30 20:40:09.444224 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.444201    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.538721 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.538690    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"]
Oct 30 20:40:09.554824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.554798    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.554986 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.554859    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.554986 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.554949    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgqf4\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-kube-api-access-hgqf4\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.555097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.555025    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.555097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.555061    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.555177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.555162    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.555216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.555198    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.555257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.555242    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.555297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.555273    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aacf197-1de8-403b-8e73-c3cfd73452a0-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656357    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656413    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656438    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgqf4\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-kube-api-access-hgqf4\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656499    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656570    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656667    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656700    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656746    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.656816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.656774    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aacf197-1de8-403b-8e73-c3cfd73452a0-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.657100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.657076    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.657267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.657245    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.657335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.657254    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.657398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.657344    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.657642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.657622    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aacf197-1de8-403b-8e73-c3cfd73452a0-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.659026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.659006    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.659410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.659387    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.715109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.715079    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgqf4\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-kube-api-access-hgqf4\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.715234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.715191    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:09.762348 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:09.762328    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:10.195561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.195516    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"]
Oct 30 20:40:10.197902 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:40:10.197871    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aacf197_1de8_403b_8e73_c3cfd73452a0.slice/crio-d10b831a398e5dd8efbbb3d08c5aab2e9bc764e94afb89a8028d84516775b49d WatchSource:0}: Error finding container d10b831a398e5dd8efbbb3d08c5aab2e9bc764e94afb89a8028d84516775b49d: Status 404 returned error can't find the container with id d10b831a398e5dd8efbbb3d08c5aab2e9bc764e94afb89a8028d84516775b49d
Oct 30 20:40:10.200458 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.200424    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:10.200566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.200492    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:10.200566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.200519    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:10.290493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.290454    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" event={"ID":"1aacf197-1de8-403b-8e73-c3cfd73452a0","Type":"ContainerStarted","Data":"5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f"}
Oct 30 20:40:10.290636 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.290586    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" event={"ID":"1aacf197-1de8-403b-8e73-c3cfd73452a0","Type":"ContainerStarted","Data":"d10b831a398e5dd8efbbb3d08c5aab2e9bc764e94afb89a8028d84516775b49d"}
Oct 30 20:40:10.416696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.416639    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podStartSLOduration=1.416621228 podStartE2EDuration="1.416621228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:10.413648066 +0000 UTC m=+1236.562699054" watchObservedRunningTime="2026-10-30 20:40:10.416621228 +0000 UTC m=+1236.565672216"
Oct 30 20:40:10.763736 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:10.763643    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:11.763863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:11.763821    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:40:11.764369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:11.763882    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:40:12.766557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:12.765916    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:40:12.766557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:12.765979    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:40:13.072128 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:13.071994    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:13.763340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:13.763310    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:40:13.763510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:13.763360    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:40:14.762864 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:14.762826    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:40:14.763280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:14.762882    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:40:15.763703 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:15.763663    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:40:15.764099 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:15.763729    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:40:16.763332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:16.763288    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:40:16.763516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:16.763352    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:40:17.766962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:17.766936    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:17.767441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:17.767423    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:17.768039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:17.768023    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:40:17.924547 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:17.924490    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:22.595325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:22.595289    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:27.274149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:27.274117    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:31.078950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:31.078913    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:32.115868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:32.115824    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:32.989330 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:32.989296    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kd87j-mbvcw"]
Oct 30 20:40:32.993688 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:32.993669    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.049997 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.049966    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kd87j-mbvcw"]
Oct 30 20:40:33.168516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.168487    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9147a14d-792a-45e5-8ea8-fe56dfc2b725-archives-path\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.168906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.168604    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-service-ca-bundle\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.168906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.168629    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbtn\" (UniqueName: \"kubernetes.io/projected/9147a14d-792a-45e5-8ea8-fe56dfc2b725-kube-api-access-rjbtn\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.168906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.168664    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-trusted-ca-bundle\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.270150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.270085    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9147a14d-792a-45e5-8ea8-fe56dfc2b725-archives-path\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.270150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.270146    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-service-ca-bundle\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.270335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.270165    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbtn\" (UniqueName: \"kubernetes.io/projected/9147a14d-792a-45e5-8ea8-fe56dfc2b725-kube-api-access-rjbtn\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.270335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.270204    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-trusted-ca-bundle\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.270522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.270498    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9147a14d-792a-45e5-8ea8-fe56dfc2b725-archives-path\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.270828 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.270812    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-service-ca-bundle\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.271122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.271100    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-trusted-ca-bundle\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.328026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.327998    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbtn\" (UniqueName: \"kubernetes.io/projected/9147a14d-792a-45e5-8ea8-fe56dfc2b725-kube-api-access-rjbtn\") pod \"periodic-gathering-kd87j-mbvcw\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") " pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.606511 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.606440    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:40:33.984675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:33.984647    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kd87j-mbvcw"]
Oct 30 20:40:33.984818 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:40:33.984675    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9147a14d_792a_45e5_8ea8_fe56dfc2b725.slice/crio-ea38905a0a496101265f22e8ab7a268ff3c79fc21514fd69d662d7d7755a1daf WatchSource:0}: Error finding container ea38905a0a496101265f22e8ab7a268ff3c79fc21514fd69d662d7d7755a1daf: Status 404 returned error can't find the container with id ea38905a0a496101265f22e8ab7a268ff3c79fc21514fd69d662d7d7755a1daf
Oct 30 20:40:34.490456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:34.490427    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kd87j-mbvcw" event={"ID":"9147a14d-792a-45e5-8ea8-fe56dfc2b725","Type":"ContainerStarted","Data":"9cdd97813a883f9863be8f3434589bff225cc5556e6bf549c84f7959d08f9145"}
Oct 30 20:40:34.490851 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:34.490497    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kd87j-mbvcw" event={"ID":"9147a14d-792a-45e5-8ea8-fe56dfc2b725","Type":"ContainerStarted","Data":"ea38905a0a496101265f22e8ab7a268ff3c79fc21514fd69d662d7d7755a1daf"}
Oct 30 20:40:34.607783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:34.607736    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kd87j-mbvcw" podStartSLOduration=2.607723101 podStartE2EDuration="2.607723101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:34.606975843 +0000 UTC m=+1260.756026832" watchObservedRunningTime="2026-10-30 20:40:34.607723101 +0000 UTC m=+1260.756774087"
Oct 30 20:40:36.776920 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:36.776882    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:37.114707 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:40:37.114626    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:40:37.475255 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.475225    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"]
Oct 30 20:40:37.483643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.483621    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.524542 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.524510    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"]
Oct 30 20:40:37.529132 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.529096    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.539453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.539429    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-df031b-openshift-default-dockercfg-8m6p4\""
Oct 30 20:40:37.621936 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.621907    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.621965    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.622026    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda145c-244b-4d9b-9888-06ced80ac5ef-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.622134    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.622158    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.622190    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lh4\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-kube-api-access-65lh4\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.622215    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.622233    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.622447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.622254    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.630694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.630668    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"]
Oct 30 20:40:37.700238 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.700214    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"]
Oct 30 20:40:37.723578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723521    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.723720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723598    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.723720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723646    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.723720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723673    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.723720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723705    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.723928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723752    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lh4\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-kube-api-access-65lh4\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.723928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723775    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.723928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723818    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.723928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723848    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.723928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723884    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.723928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723915    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.724215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.723992    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.724215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.724023    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.724215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.724084    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnzg\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-kube-api-access-bjnzg\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.724215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.724112    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.724215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.724140    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda145c-244b-4d9b-9888-06ced80ac5ef-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.724215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.724162    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.724524 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.724239    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.724790 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.724768    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.725152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.725125    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.725623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.725569    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.726502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.726080    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.726502 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.726254    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda145c-244b-4d9b-9888-06ced80ac5ef-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.736583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.728728    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.737114 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.737090    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.781698 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.781677    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.781979 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.781815    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lh4\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-kube-api-access-65lh4\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.798160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.798126    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:37.825338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825307    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825381    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825442    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825504    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnzg\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-kube-api-access-bjnzg\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825551    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825611    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825676    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825717    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825766    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.825962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.825805    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.826069 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.826049    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.826128 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.826111    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.826717 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.826682    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.826956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.826931    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.828454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.828425    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.828454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.828443    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.886661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.886602    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:37.889211 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:37.889153    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnzg\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-kube-api-access-bjnzg\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:38.145668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.145593    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:38.233154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.232987    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"]
Oct 30 20:40:38.235110 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:40:38.235078    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dda145c_244b_4d9b_9888_06ced80ac5ef.slice/crio-705d39bb7b0882ad35988a651c878e6ef52c3fcdf3a537240a012d993c27421e WatchSource:0}: Error finding container 705d39bb7b0882ad35988a651c878e6ef52c3fcdf3a537240a012d993c27421e: Status 404 returned error can't find the container with id 705d39bb7b0882ad35988a651c878e6ef52c3fcdf3a537240a012d993c27421e
Oct 30 20:40:38.237246 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.237207    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.237342 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.237293    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.237342 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.237337    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.521375 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.521338    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" event={"ID":"4dda145c-244b-4d9b-9888-06ced80ac5ef","Type":"ContainerStarted","Data":"29dcdd9f0baabf0b3dfc7659d386573f69d554376d18f3ae74be4657a9c6fe56"}
Oct 30 20:40:38.521564 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.521405    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" event={"ID":"4dda145c-244b-4d9b-9888-06ced80ac5ef","Type":"ContainerStarted","Data":"705d39bb7b0882ad35988a651c878e6ef52c3fcdf3a537240a012d993c27421e"}
Oct 30 20:40:38.530787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.530664    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"]
Oct 30 20:40:38.535387 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:40:38.535365    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffdbee5_8fe7_4e9f_8eb2_4ebe76507b71.slice/crio-92d4b35edde33c2a5edb2fbc4588d7630779cb4a01666ea89f86fcb1f780e5b2 WatchSource:0}: Error finding container 92d4b35edde33c2a5edb2fbc4588d7630779cb4a01666ea89f86fcb1f780e5b2: Status 404 returned error can't find the container with id 92d4b35edde33c2a5edb2fbc4588d7630779cb4a01666ea89f86fcb1f780e5b2
Oct 30 20:40:38.537868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.537830    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.537958 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.537909    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.537958 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.537948    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.675632 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.675582    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podStartSLOduration=1.6755701250000001 podStartE2EDuration="1.675570125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:38.672468271 +0000 UTC m=+1264.821519284" watchObservedRunningTime="2026-10-30 20:40:38.675570125 +0000 UTC m=+1264.824621111"
Oct 30 20:40:38.799150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:38.799080    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:39.558269 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.558235    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" event={"ID":"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71","Type":"ContainerStarted","Data":"ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095"}
Oct 30 20:40:39.559313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.559287    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" event={"ID":"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71","Type":"ContainerStarted","Data":"92d4b35edde33c2a5edb2fbc4588d7630779cb4a01666ea89f86fcb1f780e5b2"}
Oct 30 20:40:39.676674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.676638    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"]
Oct 30 20:40:39.681068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.681048    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.748556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.735555    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8e2fc1-openshift-default-dockercfg-77c66\""
Oct 30 20:40:39.770556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.764570    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podStartSLOduration=2.7645487170000003 podStartE2EDuration="2.764548717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:39.680312863 +0000 UTC m=+1265.829363877" watchObservedRunningTime="2026-10-30 20:40:39.764548717 +0000 UTC m=+1265.913599701"
Oct 30 20:40:39.783559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.783392    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"]
Oct 30 20:40:39.800093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.800064    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 20:40:39.800442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.800110    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 20:40:39.886365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886181    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886242    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklns\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-kube-api-access-bklns\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886290    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886370    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886444    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886484    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886697    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886774    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.886917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.886845    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e7cdb7-12ab-462a-9c63-b18e2038e659-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989103    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989204    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989250    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989663    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989758    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989825    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989853    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e7cdb7-12ab-462a-9c63-b18e2038e659-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989974 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989915    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.989974 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.989965    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklns\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-kube-api-access-bklns\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.990071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.990002    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.990574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.990242    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.991149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.990670    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.991634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.991608    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.991722 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.991653    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e7cdb7-12ab-462a-9c63-b18e2038e659-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.992574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.992137    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:39.993629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:39.993608    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:40.071573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.068330    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklns\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-kube-api-access-bklns\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:40.071573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.068846    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:40.151625 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.149909    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:40.167566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.162661    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:40:40.167566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.162729    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:40:40.295435 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.295404    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:40.798545 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.798501    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 20:40:40.798796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.798577    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 20:40:40.843034 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.842992    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"]
Oct 30 20:40:40.864568 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:40:40.853805    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e7cdb7_12ab_462a_9c63_b18e2038e659.slice/crio-ff6ae7025ea4a6d083f9a1fb1d4de49b8dfa2bbacff76f74bc923864c89bba02 WatchSource:0}: Error finding container ff6ae7025ea4a6d083f9a1fb1d4de49b8dfa2bbacff76f74bc923864c89bba02: Status 404 returned error can't find the container with id ff6ae7025ea4a6d083f9a1fb1d4de49b8dfa2bbacff76f74bc923864c89bba02
Oct 30 20:40:40.872552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.869257    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:40.872821 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.872795    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:40.872958 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:40.872941    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:41.148618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.147966    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:40:41.148618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.148042    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:40:41.591711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.591669    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" event={"ID":"55e7cdb7-12ab-462a-9c63-b18e2038e659","Type":"ContainerStarted","Data":"2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861"}
Oct 30 20:40:41.591863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.591734    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" event={"ID":"55e7cdb7-12ab-462a-9c63-b18e2038e659","Type":"ContainerStarted","Data":"ff6ae7025ea4a6d083f9a1fb1d4de49b8dfa2bbacff76f74bc923864c89bba02"}
Oct 30 20:40:41.717554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.716482    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podStartSLOduration=2.716463882 podStartE2EDuration="2.716463882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:41.714796866 +0000 UTC m=+1267.863847855" watchObservedRunningTime="2026-10-30 20:40:41.716463882 +0000 UTC m=+1267.865514870"
Oct 30 20:40:41.775559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.775059    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:41.799316 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.799285    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 20:40:41.799489 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:41.799332    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 20:40:42.150567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:42.148426    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:40:42.150567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:42.148499    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:40:42.303790 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:42.303744    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:42.305359 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:42.305333    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:40:42.305471 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:42.305380    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:40:42.799555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:42.799110    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 20:40:42.799555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:42.799170    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 20:40:43.151617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:43.147822    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:40:43.151617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:43.147877    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:40:43.296929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:43.296884    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:40:43.297123 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:43.296947    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:40:43.801556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:43.800882    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 20:40:43.801556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:43.800942    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 20:40:44.147413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:44.146911    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:40:44.147413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:44.146975    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:40:44.227558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:44.209331    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:44.300556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:44.298903    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:40:44.300556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:44.298978    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:40:44.801918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:44.799259    5002 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 20:40:44.801918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:44.799330    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 20:40:45.160063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:45.159649    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:40:45.160063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:45.159733    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:40:45.299554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:45.298825    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:40:45.299554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:45.298883    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:40:46.335557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.321802    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:40:46.335557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.321859    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:40:46.571560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.571376    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:46.572670 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.572634    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:46.573068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.572989    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:46.573390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.573370    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:46.573492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.573476    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:40:46.574186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.574171    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:40:46.707594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:46.707141    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:47.296801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:47.296765    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:40:47.296974 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:47.296825    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:40:48.128383 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:48.128341    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:40:48.296930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:48.296900    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:40:48.297091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:48.296956    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:40:50.344558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:50.338620    5002 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:40:50.344558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:50.338708    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:40:50.849773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:50.849734    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:50.850410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:50.850367    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:50.851509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:50.851473    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:40:51.549284 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:51.549234    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:56.208498 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:40:56.208458    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:01.132813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:01.132756    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:04.645255 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:04.645214    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:05.125178 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:05.125142    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:05.914625 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:05.914597    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:07.115718 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:41:07.115679    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:41:10.844444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:10.844408    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:12.993454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:12.993381    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"]
Oct 30 20:41:12.993858 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:12.993622    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy" containerID="cri-o://e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a" gracePeriod=30
Oct 30 20:41:15.861431 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:15.861342    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:18.274746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.274723    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:41:18.423172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423138    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istiod-ca-cert\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423187    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-socket\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423223    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-credential-socket\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423280    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-envoy\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423351    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-data\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423413    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-podinfo\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423498    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-socket" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:18.423617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423510    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-token\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-token\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423599    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-data" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:18.423772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423637    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-certs\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423709    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-kube-api-access-rrhc8\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-kube-api-access-rrhc8\") pod \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\" (UID: \"6fae8f0e-17a6-4689-8da3-7702b3e4842e\") "
Oct 30 20:41:18.423772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423734    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istiod-ca-cert" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:18.423946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423918    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-credential-socket" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:18.423946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.423932    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-certs" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:18.424275 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.424242    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.424275 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.424267    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.424412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.424281    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.424412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.424293    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.424412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.424305    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.425746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.425720    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-token" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:18.426170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.426148    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-envoy" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:18.426362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.426339    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-podinfo" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:41:18.426442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.426425    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-kube-api-access-rrhc8" pod "6fae8f0e-17a6-4689-8da3-7702b3e4842e" (UID: "6fae8f0e-17a6-4689-8da3-7702b3e4842e"). InnerVolumeSpecName "kube-api-access-rrhc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:18.525442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.525366    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.525442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.525394    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.525442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.525403    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrhc8\" (UniqueName: \"kubernetes.io/projected/6fae8f0e-17a6-4689-8da3-7702b3e4842e-kube-api-access-rrhc8\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.525442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:18.525413    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fae8f0e-17a6-4689-8da3-7702b3e4842e-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:19.006108 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.006078    5002 generic.go:356] "Generic (PLEG): container finished" podID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerID="e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a" exitCode=0
Oct 30 20:41:19.006281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.006123    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"
Oct 30 20:41:19.006281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.006167    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" event={"ID":"6fae8f0e-17a6-4689-8da3-7702b3e4842e","Type":"ContainerDied","Data":"e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a"}
Oct 30 20:41:19.006281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.006219    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx" event={"ID":"6fae8f0e-17a6-4689-8da3-7702b3e4842e","Type":"ContainerDied","Data":"362131bbbc9bf83ff28c1cdf9e75c7cb50a887fc6bd40f7ed24349ca9a51e0d7"}
Oct 30 20:41:19.006281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.006247    5002 scope.go:122] "RemoveContainer" containerID="e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a"
Oct 30 20:41:19.019356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.019339    5002 scope.go:122] "RemoveContainer" containerID="e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a"
Oct 30 20:41:19.019616 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:41:19.019597    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a\": container with ID starting with e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a not found: ID does not exist" containerID="e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a"
Oct 30 20:41:19.019687 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.019623    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a"} err="failed to get container status \"e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a\": rpc error: code = NotFound desc = could not find container \"e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a\": container with ID starting with e2c7c16559d0a8fc103bf413022ce6d18090d8f46d38512bbeef58fd97fc159a not found: ID does not exist"
Oct 30 20:41:19.174208 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.174180    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"]
Oct 30 20:41:19.223677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:19.223634    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx"]
Oct 30 20:41:20.010561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:20.010481    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" path="/var/lib/kubelet/pods/6fae8f0e-17a6-4689-8da3-7702b3e4842e/volumes"
Oct 30 20:41:20.936558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:20.936024    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:22.338658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:22.338616    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.538648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:22.538601    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.514578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:23.432103    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 20:41:23.514578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:23.470655    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:41:23.514578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:23.478980    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.707463 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:23.707421    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 20:41:23.756598 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:23.738544    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:41:23.763559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:23.756834    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:41:24.109122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:24.108522    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:41:24.109122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:24.108596    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:41:26.148664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:26.148618    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:27.167629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.167560    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-2ec1ea"]
Oct 30 20:41:27.168136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.168122    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy"
Oct 30 20:41:27.168182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.168138    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy"
Oct 30 20:41:27.168234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.168225    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fae8f0e-17a6-4689-8da3-7702b3e4842e" containerName="istio-proxy"
Oct 30 20:41:27.171912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.171891    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:27.228609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.228584    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-2ec1ea"]
Oct 30 20:41:27.274014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.273993    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vjnsm\""
Oct 30 20:41:27.312513 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.312492    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hgp\" (UniqueName: \"kubernetes.io/projected/77d47386-09bc-4883-bcd5-4d392624bc78-kube-api-access-74hgp\") pod \"test-curl-7269-2ec1ea\" (UID: \"77d47386-09bc-4883-bcd5-4d392624bc78\") " pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:27.413262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.413237    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hgp\" (UniqueName: \"kubernetes.io/projected/77d47386-09bc-4883-bcd5-4d392624bc78-kube-api-access-74hgp\") pod \"test-curl-7269-2ec1ea\" (UID: \"77d47386-09bc-4883-bcd5-4d392624bc78\") " pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:27.471487 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.471461    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hgp\" (UniqueName: \"kubernetes.io/projected/77d47386-09bc-4883-bcd5-4d392624bc78-kube-api-access-74hgp\") pod \"test-curl-7269-2ec1ea\" (UID: \"77d47386-09bc-4883-bcd5-4d392624bc78\") " pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:27.484316 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.484298    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:27.870868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.870841    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-2ec1ea"]
Oct 30 20:41:27.871590 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:27.871566    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77d47386_09bc_4883_bcd5_4d392624bc78.slice/crio-351811ede354db86d077ceb638110394543d969267b4cc33b36b90a0bbd26907 WatchSource:0}: Error finding container 351811ede354db86d077ceb638110394543d969267b4cc33b36b90a0bbd26907: Status 404 returned error can't find the container with id 351811ede354db86d077ceb638110394543d969267b4cc33b36b90a0bbd26907
Oct 30 20:41:27.873756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:27.873737    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:41:28.794883 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:28.794850    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-2ec1ea" event={"ID":"77d47386-09bc-4883-bcd5-4d392624bc78","Type":"ContainerStarted","Data":"351811ede354db86d077ceb638110394543d969267b4cc33b36b90a0bbd26907"}
Oct 30 20:41:30.813276 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:30.813175    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-2ec1ea" event={"ID":"77d47386-09bc-4883-bcd5-4d392624bc78","Type":"ContainerStarted","Data":"1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c"}
Oct 30 20:41:30.934724 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:30.934659    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-2ec1ea" podStartSLOduration=1.328312452 podStartE2EDuration="3.934641508s" totalImagesPullingTime="2.606329056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:30.92903927 +0000 UTC m=+1317.078090254" watchObservedRunningTime="2026-10-30 20:41:30.934641508 +0000 UTC m=+1317.083692499"
Oct 30 20:41:31.222933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:31.222874    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:31.411523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:31.411490    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:32.477078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:32.477047    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-2ec1ea"]
Oct 30 20:41:32.481329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:32.481304    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-2ec1ea"]
Oct 30 20:41:32.826657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:32.826565    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-2ec1ea" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" containerName="test-curl-7269-2ec1ea" containerID="cri-o://1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c" gracePeriod=2
Oct 30 20:41:32.878152 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:41:32.878120    5002 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-2ec1ea\" is forbidden: User \"system:node:ip-10-0-141-29.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-29.us-west-1.compute.internal' and this object" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:34.014650 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:41:34.014614    5002 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-2ec1ea\" is forbidden: User \"system:node:ip-10-0-141-29.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-29.us-west-1.compute.internal' and this object" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:34.990565 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:34.990464    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:35.100938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.100879    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:35.185001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.184964    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:35.204521 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.204495    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d47386-09bc-4883-bcd5-4d392624bc78-kube-api-access-74hgp\" (UniqueName: \"kubernetes.io/projected/77d47386-09bc-4883-bcd5-4d392624bc78-kube-api-access-74hgp\") pod \"77d47386-09bc-4883-bcd5-4d392624bc78\" (UID: \"77d47386-09bc-4883-bcd5-4d392624bc78\") "
Oct 30 20:41:35.206676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.206651    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d47386-09bc-4883-bcd5-4d392624bc78-kube-api-access-74hgp" pod "77d47386-09bc-4883-bcd5-4d392624bc78" (UID: "77d47386-09bc-4883-bcd5-4d392624bc78"). InnerVolumeSpecName "kube-api-access-74hgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:35.307235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.306715    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74hgp\" (UniqueName: \"kubernetes.io/projected/77d47386-09bc-4883-bcd5-4d392624bc78-kube-api-access-74hgp\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:35.812063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.811999    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:35.848453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.848422    5002 generic.go:356] "Generic (PLEG): container finished" podID="77d47386-09bc-4883-bcd5-4d392624bc78" containerID="1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c" exitCode=137
Oct 30 20:41:35.848658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.848467    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:35.848658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.848511    5002 scope.go:122] "RemoveContainer" containerID="1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c"
Oct 30 20:41:35.859929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.859887    5002 scope.go:122] "RemoveContainer" containerID="1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c"
Oct 30 20:41:35.860232 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:41:35.860215    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c\": container with ID starting with 1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c not found: ID does not exist" containerID="1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c"
Oct 30 20:41:35.860303 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.860242    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c"} err="failed to get container status \"1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c\": rpc error: code = NotFound desc = could not find container \"1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c\": container with ID starting with 1776125d9da29f46b11d47f3f0a5ef64162b929e847797dd635f256e0693c15c not found: ID does not exist"
Oct 30 20:41:35.914675 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:41:35.914633    5002 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-2ec1ea\" is forbidden: User \"system:node:ip-10-0-141-29.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-29.us-west-1.compute.internal' and this object" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" pod="opendatahub/test-curl-7269-2ec1ea"
Oct 30 20:41:35.967326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:35.967290    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" path="/var/lib/kubelet/pods/77d47386-09bc-4883-bcd5-4d392624bc78/volumes"
Oct 30 20:41:36.011229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:36.011193    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:37.116915 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:41:37.116880    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:41:38.699046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:38.699017    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:38.898658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:38.898627    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:39.194612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.194582    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:39.627958 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.627931    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"]
Oct 30 20:41:39.628501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.628488    5002 cpu_manager.go:414] "containerMap: removing container" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" containerName="test-curl-7269-2ec1ea"
Oct 30 20:41:39.628562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.628503    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" containerName="test-curl-7269-2ec1ea"
Oct 30 20:41:39.628636 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.628626    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d47386-09bc-4883-bcd5-4d392624bc78" containerName="test-curl-7269-2ec1ea"
Oct 30 20:41:39.633321 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.633302    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.694634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.694605    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7f0ceebd-gw-openshift-default-dockercfg-q5slv\""
Oct 30 20:41:39.750619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.750585    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"]
Oct 30 20:41:39.758488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758459    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xvg\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-kube-api-access-c5xvg\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.758709 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758503    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istiod-ca-cert\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.758709 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758623    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-socket\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.758839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758713    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-token\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.758839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758742    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-podinfo\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.758839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758787    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-credential-socket\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.758839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758816    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-certs\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.759044 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758839    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-envoy\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.759044 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.758879    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-data\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.799430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.799405    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:39.859624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859598    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xvg\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-kube-api-access-c5xvg\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859741 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859628    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istiod-ca-cert\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859741 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859665    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-socket\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859741 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859725    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-token\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859752    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-podinfo\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859791    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-credential-socket\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859821    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-certs\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859841    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-envoy\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.859919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.859874    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-data\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.860175 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.860151    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-socket\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.860242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.860196    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-credential-socket\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.860350 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.860328    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-certs\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.860426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.860379    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-data\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.860469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.860439    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istiod-ca-cert\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.862216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.862194    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-envoy\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.862389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.862371    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-podinfo\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.921365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.921292    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xvg\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-kube-api-access-c5xvg\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.921954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.921931    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-token\") pod \"e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") " pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:39.946541 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:39.946514    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:40.144555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.137039    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:40.303671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.303635    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"]
Oct 30 20:41:40.310018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.309990    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376456    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376584    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376618    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfdw\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-kube-api-access-wrfdw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376655    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376685    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376719    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376759    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376810    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.377045 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.376889    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.379284 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.379232    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"]
Oct 30 20:41:40.453305 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.453265    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"]
Oct 30 20:41:40.479551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479245    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479349    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479478    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479583    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479614    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfdw\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-kube-api-access-wrfdw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479652    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479681    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479718    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.479848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.479762    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.487606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.480235    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.487606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.483084    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.487606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.483719    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.487606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.483969    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.487606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.484197    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.487606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.485015    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.488046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.487822    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.491441 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:40.491418    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ffc7a3_1190_4dd1_9723_c7821df2fba4.slice/crio-fdb8d465fe6b24ea62dbbb35adcfaa7daa5801edcf2addcfa4771a16c046e4ad WatchSource:0}: Error finding container fdb8d465fe6b24ea62dbbb35adcfaa7daa5801edcf2addcfa4771a16c046e4ad: Status 404 returned error can't find the container with id fdb8d465fe6b24ea62dbbb35adcfaa7daa5801edcf2addcfa4771a16c046e4ad
Oct 30 20:41:40.493935 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.493902    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:40.494017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.493974    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:40.494017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.494003    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:40.543495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.543393    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfdw\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-kube-api-access-wrfdw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.549254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.549203    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.630140 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.630105    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:40.635024 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.635005    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:40.824690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.824609    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:40.896929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.896896    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" event={"ID":"46ffc7a3-1190-4dd1-9723-c7821df2fba4","Type":"ContainerStarted","Data":"42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381"}
Oct 30 20:41:40.897113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.896962    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" event={"ID":"46ffc7a3-1190-4dd1-9723-c7821df2fba4","Type":"ContainerStarted","Data":"fdb8d465fe6b24ea62dbbb35adcfaa7daa5801edcf2addcfa4771a16c046e4ad"}
Oct 30 20:41:40.947461 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:40.947431    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:41.037349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.037303    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podStartSLOduration=2.0372889 podStartE2EDuration="2.0372889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:41.031045559 +0000 UTC m=+1327.180096581" watchObservedRunningTime="2026-10-30 20:41:41.0372889 +0000 UTC m=+1327.186339883"
Oct 30 20:41:41.101577 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:41.100102    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice/crio-f754bcbe28ceb761a9822e389a961f34443db1bbef0fbb4ed94b4b4a99f371c9 WatchSource:0}: Error finding container f754bcbe28ceb761a9822e389a961f34443db1bbef0fbb4ed94b4b4a99f371c9: Status 404 returned error can't find the container with id f754bcbe28ceb761a9822e389a961f34443db1bbef0fbb4ed94b4b4a99f371c9
Oct 30 20:41:41.101577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.100950    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"]
Oct 30 20:41:41.103182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.103147    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:41.103272 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.103223    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:41.103272 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.103264    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:41.358867 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.358782    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:41.735910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.735874    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:41.906914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.906877    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" event={"ID":"0e4eca8a-ec71-4ee5-b142-91786d5364e7","Type":"ContainerStarted","Data":"4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e"}
Oct 30 20:41:41.907399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.906942    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" event={"ID":"0e4eca8a-ec71-4ee5-b142-91786d5364e7","Type":"ContainerStarted","Data":"f754bcbe28ceb761a9822e389a961f34443db1bbef0fbb4ed94b4b4a99f371c9"}
Oct 30 20:41:41.940187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.940084    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:41.952451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:41.952427    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:42.090265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:42.090169    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podStartSLOduration=2.090156487 podStartE2EDuration="2.090156487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:42.088062863 +0000 UTC m=+1328.237113850" watchObservedRunningTime="2026-10-30 20:41:42.090156487 +0000 UTC m=+1328.239207553"
Oct 30 20:41:42.637862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:42.637823    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:42.639607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:42.639578    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:41:42.639746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:42.639637    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:41:42.938759 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:42.938675    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:42.940094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:42.940067    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:41:43.636342 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:43.636306    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:41:43.636766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:43.636717    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:41:43.662672 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:43.662637    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:43.871746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:43.871715    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:44.070488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:44.070460    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:44.359761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:44.359680    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:44.586594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:44.586565    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:44.640618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:44.635862    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:41:44.640618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:44.635950    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:41:44.783606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:44.783573    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:44.983488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:44.983453    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:45.648578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:45.636892    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:41:45.648578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:45.636946    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:41:45.694791 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:45.694758    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:46.559350 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:46.559297    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:46.636344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:46.636307    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:41:46.636562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:46.636401    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:41:47.361397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.361200    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:47.567338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.567302    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"]
Oct 30 20:41:47.571663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.571643    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.605793 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.605765    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzt7\" (UniqueName: \"kubernetes.io/projected/26ec05a9-b960-4528-a9f0-2133f5a6ae00-kube-api-access-2fzt7\") pod \"payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.605975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.605952    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec05a9-b960-4528-a9f0-2133f5a6ae00-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.610866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.610842    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"]
Oct 30 20:41:47.615061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.615010    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:47.629923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.629887    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:47.630478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.630461    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"]
Oct 30 20:41:47.636356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.636006    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:41:47.636356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.636049    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:41:47.671037 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.670852    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7f0ceebd-dockercfg-grphv\""
Oct 30 20:41:47.671037 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.670941    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7f0ceebd\""
Oct 30 20:41:47.695554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.687906    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"]
Oct 30 20:41:47.754658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.747938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzt7\" (UniqueName: \"kubernetes.io/projected/26ec05a9-b960-4528-a9f0-2133f5a6ae00-kube-api-access-2fzt7\") pod \"payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.754658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.748129    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec05a9-b960-4528-a9f0-2133f5a6ae00-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.754658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.754074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec05a9-b960-4528-a9f0-2133f5a6ae00-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.810598 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.810493    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzt7\" (UniqueName: \"kubernetes.io/projected/26ec05a9-b960-4528-a9f0-2133f5a6ae00-kube-api-access-2fzt7\") pod \"payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.850252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.850210    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1bf291-d004-4bd8-b8bd-438959bf29de-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:47.850445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.850281    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbxb\" (UniqueName: \"kubernetes.io/projected/fe1bf291-d004-4bd8-b8bd-438959bf29de-kube-api-access-xwbxb\") pod \"payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:47.887901 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.887826    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:47.951475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.951432    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1bf291-d004-4bd8-b8bd-438959bf29de-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:47.951715 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.951521    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbxb\" (UniqueName: \"kubernetes.io/projected/fe1bf291-d004-4bd8-b8bd-438959bf29de-kube-api-access-xwbxb\") pod \"payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:47.953056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:47.953030    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1bf291-d004-4bd8-b8bd-438959bf29de-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:48.040738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:48.040675    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbxb\" (UniqueName: \"kubernetes.io/projected/fe1bf291-d004-4bd8-b8bd-438959bf29de-kube-api-access-xwbxb\") pod \"payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:48.228586 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:48.228512    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:48.297828 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:48.297799    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"]
Oct 30 20:41:48.302365 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:48.302163    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ec05a9_b960_4528_a9f0_2133f5a6ae00.slice/crio-1146aa9502eff8c39c16ceb1ca95d1ea7ef356b214c23980c617ced11c4204f4 WatchSource:0}: Error finding container 1146aa9502eff8c39c16ceb1ca95d1ea7ef356b214c23980c617ced11c4204f4: Status 404 returned error can't find the container with id 1146aa9502eff8c39c16ceb1ca95d1ea7ef356b214c23980c617ced11c4204f4
Oct 30 20:41:48.634973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:48.634945    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"]
Oct 30 20:41:48.636163 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:48.636136    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1bf291_d004_4bd8_b8bd_438959bf29de.slice/crio-c303df5b05a060cbb0e2dcf4e5747ff349457fa273648e4305fcaa2128b22605 WatchSource:0}: Error finding container c303df5b05a060cbb0e2dcf4e5747ff349457fa273648e4305fcaa2128b22605: Status 404 returned error can't find the container with id c303df5b05a060cbb0e2dcf4e5747ff349457fa273648e4305fcaa2128b22605
Oct 30 20:41:48.641302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:48.641272    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:48.642385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:48.642366    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:48.643018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:48.643003    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:41:49.017972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.017934    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg" event={"ID":"26ec05a9-b960-4528-a9f0-2133f5a6ae00","Type":"ContainerStarted","Data":"0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b"}
Oct 30 20:41:49.018139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.017993    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg" event={"ID":"26ec05a9-b960-4528-a9f0-2133f5a6ae00","Type":"ContainerStarted","Data":"1146aa9502eff8c39c16ceb1ca95d1ea7ef356b214c23980c617ced11c4204f4"}
Oct 30 20:41:49.018139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.018023    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:41:49.020230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.020204    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz" event={"ID":"fe1bf291-d004-4bd8-b8bd-438959bf29de","Type":"ContainerStarted","Data":"5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7"}
Oct 30 20:41:49.020362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.020253    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz" event={"ID":"fe1bf291-d004-4bd8-b8bd-438959bf29de","Type":"ContainerStarted","Data":"c303df5b05a060cbb0e2dcf4e5747ff349457fa273648e4305fcaa2128b22605"}
Oct 30 20:41:49.020362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.020279    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:41:49.138320 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.138263    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg" podStartSLOduration=2.138248579 podStartE2EDuration="2.138248579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:49.136498273 +0000 UTC m=+1335.285549260" watchObservedRunningTime="2026-10-30 20:41:49.138248579 +0000 UTC m=+1335.287299566"
Oct 30 20:41:49.259758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:49.259037    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz" podStartSLOduration=2.259019858 podStartE2EDuration="2.259019858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:49.255700101 +0000 UTC m=+1335.404751098" watchObservedRunningTime="2026-10-30 20:41:49.259019858 +0000 UTC m=+1335.408070853"
Oct 30 20:41:50.892383 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:50.892352    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:51.093985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:51.093947    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:52.175370 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.175337    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:52.361801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.361768    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:52.490596 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.488778    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"]
Oct 30 20:41:52.493861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.493833    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.536908 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.536877    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"]
Oct 30 20:41:52.540826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.540808    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.594758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.594737    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"]
Oct 30 20:41:52.618700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.618674    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.618808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.618701    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.618808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.618730    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.618808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.618790    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.618950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.618872    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76c2v\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-kube-api-access-76c2v\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.618950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.618900    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.619051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.618981    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.619051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619021    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.619129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619070    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8w6\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-kube-api-access-fq8w6\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.619129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619121    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.619201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619140    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c790f12-d632-4162-85f1-8222c41b4d3f-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.619201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619167    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.619265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619213    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.619265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619240    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.619341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619329    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.619421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619408    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.619501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619477    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.619554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.619524    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.720557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.720127    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"]
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721034    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721080    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721113    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721149    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721176    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721210    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721242    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721273    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76c2v\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-kube-api-access-76c2v\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721321    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721365    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721389    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721419    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8w6\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-kube-api-access-fq8w6\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.721498    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.727280    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"]
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.732483    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c790f12-d632-4162-85f1-8222c41b4d3f-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.732621    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.732666    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.733382    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.733671    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.734267    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.734858    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.735567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.735140    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.736832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.736819    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.781677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.737214    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.781677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.738226    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.781677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.739742    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.781677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.743511    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.781677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.743793    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c790f12-d632-4162-85f1-8222c41b4d3f-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.798253 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.794967    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.799255 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.799227    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:52.801090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.801047    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.801572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.801513    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.803580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.803064    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.803580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.803210    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.803580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.803271    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.803807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.803724    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"]
Oct 30 20:41:52.803867 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.803820    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.803948 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.803928    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76c2v\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-kube-api-access-76c2v\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.804088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.804070    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8w6\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-kube-api-access-fq8w6\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.810258 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.810238    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:52.852546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.852514    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8660b337-dockercfg-9b64s\""
Oct 30 20:41:52.859704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.859678    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8660b337\""
Oct 30 20:41:52.860954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.860927    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:52.937012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.936974    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2deb2c54-6bd4-4a9f-843e-c799db781430-plugins-config-volume\") pod \"payload-processing-e2e-ait-8660b337-f578c795b-z45vn\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") " pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:52.937429 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:52.937410    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7c7p\" (UniqueName: \"kubernetes.io/projected/2deb2c54-6bd4-4a9f-843e-c799db781430-kube-api-access-n7c7p\") pod \"payload-processing-e2e-ait-8660b337-f578c795b-z45vn\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") " pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:53.018044 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.018004    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:53.039145 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.039049    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2deb2c54-6bd4-4a9f-843e-c799db781430-plugins-config-volume\") pod \"payload-processing-e2e-ait-8660b337-f578c795b-z45vn\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") " pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:53.039754 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.039729    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7c7p\" (UniqueName: \"kubernetes.io/projected/2deb2c54-6bd4-4a9f-843e-c799db781430-kube-api-access-n7c7p\") pod \"payload-processing-e2e-ait-8660b337-f578c795b-z45vn\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") " pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:53.041673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.041651    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2deb2c54-6bd4-4a9f-843e-c799db781430-plugins-config-volume\") pod \"payload-processing-e2e-ait-8660b337-f578c795b-z45vn\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") " pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:53.105078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.105050    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7c7p\" (UniqueName: \"kubernetes.io/projected/2deb2c54-6bd4-4a9f-843e-c799db781430-kube-api-access-n7c7p\") pod \"payload-processing-e2e-ait-8660b337-f578c795b-z45vn\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") " pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:53.193701 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.193672    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:53.277801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.277773    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"]
Oct 30 20:41:53.304898 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:53.304866    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c790f12_d632_4162_85f1_8222c41b4d3f.slice/crio-61a47037cbbfe469223e4fa156881838c7c8fc6549b886a8a5d3b78e9e385b8e WatchSource:0}: Error finding container 61a47037cbbfe469223e4fa156881838c7c8fc6549b886a8a5d3b78e9e385b8e: Status 404 returned error can't find the container with id 61a47037cbbfe469223e4fa156881838c7c8fc6549b886a8a5d3b78e9e385b8e
Oct 30 20:41:53.307929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.307885    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.308017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.307982    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.308058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.308030    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.341564 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.341505    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"]
Oct 30 20:41:53.348977 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:53.348943    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a838c0f_9a0d_4c8b_b937_0961fbb5b843.slice/crio-33b106e0830df59b2a6c64217dfde115e91c2403dd9d7fb6412abff27ecc6c15 WatchSource:0}: Error finding container 33b106e0830df59b2a6c64217dfde115e91c2403dd9d7fb6412abff27ecc6c15: Status 404 returned error can't find the container with id 33b106e0830df59b2a6c64217dfde115e91c2403dd9d7fb6412abff27ecc6c15
Oct 30 20:41:53.352600 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.352398    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.352600 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.352487    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.352600 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.352564    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.587014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:53.586984    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"]
Oct 30 20:41:53.587616 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:53.587586    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deb2c54_6bd4_4a9f_843e_c799db781430.slice/crio-1a55b729f9f48c84068f2dcf29ecb3576a3fd1f13f739f920ba9bfb9a694ab28 WatchSource:0}: Error finding container 1a55b729f9f48c84068f2dcf29ecb3576a3fd1f13f739f920ba9bfb9a694ab28: Status 404 returned error can't find the container with id 1a55b729f9f48c84068f2dcf29ecb3576a3fd1f13f739f920ba9bfb9a694ab28
Oct 30 20:41:54.084264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.083874    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn" event={"ID":"2deb2c54-6bd4-4a9f-843e-c799db781430","Type":"ContainerStarted","Data":"db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b"}
Oct 30 20:41:54.084264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.083936    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn" event={"ID":"2deb2c54-6bd4-4a9f-843e-c799db781430","Type":"ContainerStarted","Data":"1a55b729f9f48c84068f2dcf29ecb3576a3fd1f13f739f920ba9bfb9a694ab28"}
Oct 30 20:41:54.084264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.083974    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:41:54.086807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.086774    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" event={"ID":"4a838c0f-9a0d-4c8b-b937-0961fbb5b843","Type":"ContainerStarted","Data":"cbe6deb35ae963f380b419ae3b92156018fef5c9d9c8c692f0a9d8ce80660cbe"}
Oct 30 20:41:54.086934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.086843    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" event={"ID":"4a838c0f-9a0d-4c8b-b937-0961fbb5b843","Type":"ContainerStarted","Data":"33b106e0830df59b2a6c64217dfde115e91c2403dd9d7fb6412abff27ecc6c15"}
Oct 30 20:41:54.088600 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.088577    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" event={"ID":"3c790f12-d632-4162-85f1-8222c41b4d3f","Type":"ContainerStarted","Data":"e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543"}
Oct 30 20:41:54.088715 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.088631    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" event={"ID":"3c790f12-d632-4162-85f1-8222c41b4d3f","Type":"ContainerStarted","Data":"61a47037cbbfe469223e4fa156881838c7c8fc6549b886a8a5d3b78e9e385b8e"}
Oct 30 20:41:54.343612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.343275    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn" podStartSLOduration=2.343253427 podStartE2EDuration="2.343253427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.217687838 +0000 UTC m=+1340.366738846" watchObservedRunningTime="2026-10-30 20:41:54.343253427 +0000 UTC m=+1340.492304415"
Oct 30 20:41:54.402871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.402803    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podStartSLOduration=2.402788551 podStartE2EDuration="2.402788551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.364716814 +0000 UTC m=+1340.513767801" watchObservedRunningTime="2026-10-30 20:41:54.402788551 +0000 UTC m=+1340.551839539"
Oct 30 20:41:54.501396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.501338    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podStartSLOduration=2.501325613 podStartE2EDuration="2.501325613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.497374171 +0000 UTC m=+1340.646425159" watchObservedRunningTime="2026-10-30 20:41:54.501325613 +0000 UTC m=+1340.650376600"
Oct 30 20:41:54.756558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.743420    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:41:54.832557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.823036    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:41:54.853710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.853668    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:41:54.853984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.853957    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:41:54.862832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.862801    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:41:54.867652 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.867620    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:41:54.867855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:54.867831    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:41:55.390592 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.390556    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"]
Oct 30 20:41:55.394929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.394907    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506666    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506733    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506778    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506845    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506894    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506926    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506972    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942ad519-417a-4b3f-be9c-5e104267f692-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.506995    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbxt\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-kube-api-access-jdbxt\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.519279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.507072    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942ad519-417a-4b3f-be9c-5e104267f692-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.594555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.551763    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"]
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.630689    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.630741    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.630772    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.630817    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942ad519-417a-4b3f-be9c-5e104267f692-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.630845    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbxt\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-kube-api-access-jdbxt\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.630921    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942ad519-417a-4b3f-be9c-5e104267f692-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.631089    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.631146    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.631188    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.631785    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.632184    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.731445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.711310    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942ad519-417a-4b3f-be9c-5e104267f692-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.756025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.755987    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.787658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.782832    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.828561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.819910    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:41:55.828561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.819964    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:41:55.828561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.827196    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbxt\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-kube-api-access-jdbxt\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.828561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.827596    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.858163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.858128    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:55.975089 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.974942    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:41:55.975089 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:55.975017    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:41:56.001233 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.001150    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942ad519-417a-4b3f-be9c-5e104267f692-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:56.165557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.161444    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:41:56.521563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.520654    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:56.813016 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.812978    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:41:56.813385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.813334    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:41:56.820820 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.820610    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:56.863026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.862992    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:41:56.863328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:56.863280    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:41:57.073615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.066631    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.073615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.067050    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.163558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.151062    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:57.163558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.151165    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.198572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.190673    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.200898 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.200866    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.322559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.297776    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.521576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.521514    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.581646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.581603    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:41:57.704903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.704860    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.723763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.723711    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.743562 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:41:57.734262    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942ad519_417a_4b3f_be9c_5e104267f692.slice/crio-a4b9bab96ebd84f91d46f6b66b4db1a2f638eb21ce17584b778bf3f48bdc1eae WatchSource:0}: Error finding container a4b9bab96ebd84f91d46f6b66b4db1a2f638eb21ce17584b778bf3f48bdc1eae: Status 404 returned error can't find the container with id a4b9bab96ebd84f91d46f6b66b4db1a2f638eb21ce17584b778bf3f48bdc1eae
Oct 30 20:41:57.762612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.762570    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:57.769962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.769928    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:57.770180 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.770156    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:57.867729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.867681    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:41:57.871557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.867984    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:41:57.930482 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.872045    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.962569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.930859    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:41:57.962569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.922020    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:41:57.962569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:57.931654    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:41:58.075635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.075508    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.078293 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.078272    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:41:58.080641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.080607    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.127558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.082268    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:41:58.258702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.098312    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:58.258702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.154500    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.258702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.236354    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.258702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.257089    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.469816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.469783    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.531557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.525978    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:58.626038 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.580671    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.637562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.634098    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"]
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.942938    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.943008    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.943046    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.943080    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.726647    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.906121    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.952795    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.906478    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.952848    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.909262    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.031577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:58.954766    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:41:59.074568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.060273    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 20:41:59.074568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.064612    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" event={"ID":"942ad519-417a-4b3f-be9c-5e104267f692","Type":"ContainerStarted","Data":"a4b9bab96ebd84f91d46f6b66b4db1a2f638eb21ce17584b778bf3f48bdc1eae"}
Oct 30 20:41:59.082680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.078833    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.172834 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.169100    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.226707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.226626    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.236565 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.236475    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.262706 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.262671    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.462500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.462260    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:41:59.479379 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.479338    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.645475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.622900    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:41:59.645475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.628823    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.721678 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.706796    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 20:41:59.741724 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.740858    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:41:59.926914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.925918    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.926914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:41:59.925969    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:42:00.113608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.096771    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:42:00.113608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.096866    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:42:00.235003 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.216092    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.295699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.295651    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.365754 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.351803    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.531557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.527642    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.680765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.680664    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.881639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.846889    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:42:00.881639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.846961    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:42:00.934939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:00.934816    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:01.005228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.005194    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.005480 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.005453    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:42:01.008278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.008224    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podStartSLOduration=6.008209542 podStartE2EDuration="6.008209542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:00.937421481 +0000 UTC m=+1347.086472469" watchObservedRunningTime="2026-10-30 20:42:01.008209542 +0000 UTC m=+1347.157260533"
Oct 30 20:42:01.080407 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:42:01.080370    5002 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.101s"
Oct 30 20:42:01.090700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.090663    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" event={"ID":"942ad519-417a-4b3f-be9c-5e104267f692","Type":"ContainerStarted","Data":"d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef"}
Oct 30 20:42:01.091058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.091037    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:42:01.091679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.091657    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:42:01.091929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.091916    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:42:01.092147 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.092136    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:42:01.092364 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.092352    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 20:42:01.092595 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.092582    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:42:01.105877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.102146    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:42:01.165100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.165069    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:42:01.166982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.166748    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.166982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.166828    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 20:42:01.463465 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.463428    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:01.680569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.679727    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:01.828634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.813852    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.828634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.813907    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:42:01.864571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.861870    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.864571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:01.861912    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:42:02.165407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:02.165303    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 20:42:02.165626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:02.165385    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 20:42:02.765476 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:02.765442    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:02.979754 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:02.979718    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:42:02.980909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:02.980887    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:42:02.982843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:02.982827    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:42:03.030611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:03.025409    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:03.165385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:03.165351    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 20:42:03.165642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:03.165618    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 20:42:03.914457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:03.908874    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:42:03.914457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:03.908967    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:04.187615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:04.178983    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 20:42:04.187615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:04.179047    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 20:42:04.639660 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:04.639629    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:42:04.640982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:04.640962    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:42:04.647209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:04.647189    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:42:04.834357 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:04.834324    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:05.138518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:05.138487    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:42:05.166193 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:05.166161    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 20:42:05.166354 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:05.166230    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 20:42:06.165566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:06.165512    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 20:42:06.165948 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:06.165601    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 20:42:06.844973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:06.844929    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:07.126961 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:42:07.126855    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:42:07.513676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:07.513646    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:42:07.515577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:07.515554    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:42:07.517982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:07.517934    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:42:10.017519 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:10.017485    5002 generic.go:356] "Generic (PLEG): container finished" podID="9147a14d-792a-45e5-8ea8-fe56dfc2b725" containerID="9cdd97813a883f9863be8f3434589bff225cc5556e6bf549c84f7959d08f9145" exitCode=0
Oct 30 20:42:10.017896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:10.017572    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kd87j-mbvcw" event={"ID":"9147a14d-792a-45e5-8ea8-fe56dfc2b725","Type":"ContainerDied","Data":"9cdd97813a883f9863be8f3434589bff225cc5556e6bf549c84f7959d08f9145"}
Oct 30 20:42:11.195898 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.195874    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:42:11.303175 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.303101    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-service-ca-bundle\") pod \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") "
Oct 30 20:42:11.303324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.303187    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9147a14d-792a-45e5-8ea8-fe56dfc2b725-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9147a14d-792a-45e5-8ea8-fe56dfc2b725-archives-path\") pod \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") "
Oct 30 20:42:11.303388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.303324    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9147a14d-792a-45e5-8ea8-fe56dfc2b725-kube-api-access-rjbtn\" (UniqueName: \"kubernetes.io/projected/9147a14d-792a-45e5-8ea8-fe56dfc2b725-kube-api-access-rjbtn\") pod \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") "
Oct 30 20:42:11.303388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.303354    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-trusted-ca-bundle\") pod \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\" (UID: \"9147a14d-792a-45e5-8ea8-fe56dfc2b725\") "
Oct 30 20:42:11.303492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.303389    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-service-ca-bundle" pod "9147a14d-792a-45e5-8ea8-fe56dfc2b725" (UID: "9147a14d-792a-45e5-8ea8-fe56dfc2b725"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:11.303492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.303460    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9147a14d-792a-45e5-8ea8-fe56dfc2b725-archives-path" pod "9147a14d-792a-45e5-8ea8-fe56dfc2b725" (UID: "9147a14d-792a-45e5-8ea8-fe56dfc2b725"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:11.303736 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.303718    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-trusted-ca-bundle" pod "9147a14d-792a-45e5-8ea8-fe56dfc2b725" (UID: "9147a14d-792a-45e5-8ea8-fe56dfc2b725"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:11.304020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.304001    5002 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-service-ca-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:11.304083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.304027    5002 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9147a14d-792a-45e5-8ea8-fe56dfc2b725-archives-path\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:11.304083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.304040    5002 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147a14d-792a-45e5-8ea8-fe56dfc2b725-trusted-ca-bundle\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:11.305610 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.305586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9147a14d-792a-45e5-8ea8-fe56dfc2b725-kube-api-access-rjbtn" pod "9147a14d-792a-45e5-8ea8-fe56dfc2b725" (UID: "9147a14d-792a-45e5-8ea8-fe56dfc2b725"). InnerVolumeSpecName "kube-api-access-rjbtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:11.405252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.405218    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbtn\" (UniqueName: \"kubernetes.io/projected/9147a14d-792a-45e5-8ea8-fe56dfc2b725-kube-api-access-rjbtn\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:11.814914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:11.814879    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:12.036123 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:12.036098    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kd87j-mbvcw"
Oct 30 20:42:12.036291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:12.036102    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kd87j-mbvcw" event={"ID":"9147a14d-792a-45e5-8ea8-fe56dfc2b725","Type":"ContainerDied","Data":"ea38905a0a496101265f22e8ab7a268ff3c79fc21514fd69d662d7d7755a1daf"}
Oct 30 20:42:12.036291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:12.036214    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea38905a0a496101265f22e8ab7a268ff3c79fc21514fd69d662d7d7755a1daf"
Oct 30 20:42:13.442297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:13.442261    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:13.708251 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:13.708170    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:16.718078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:16.718043    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:19.512804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:19.512770    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:19.887673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:19.887584    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:20.116630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:20.116598    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:20.994792 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:20.994760    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:21.797469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:21.797436    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:26.367906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:26.367876    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:26.588023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:26.587990    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:27.093862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:27.093759    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:28.576377 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.576349    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:28.706603 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.706571    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"]
Oct 30 20:42:28.707202 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.707188    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9147a14d-792a-45e5-8ea8-fe56dfc2b725" containerName="insights-gathering"
Oct 30 20:42:28.707244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.707204    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9147a14d-792a-45e5-8ea8-fe56dfc2b725" containerName="insights-gathering"
Oct 30 20:42:28.707325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.707316    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9147a14d-792a-45e5-8ea8-fe56dfc2b725" containerName="insights-gathering"
Oct 30 20:42:28.712184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.712164    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.813620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.813590    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.813767 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.813704    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.813767 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.813742    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9963d70a-6261-48fc-b1b7-aa3138751293-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.813861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.813842    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.813914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.813875    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.813965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.813919    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.814021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.814001    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.814098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.814083    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9963d70a-6261-48fc-b1b7-aa3138751293-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.814160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.814142    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqk8\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-kube-api-access-wxqk8\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.829596 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.829492    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"]
Oct 30 20:42:28.914872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.914849    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.914903    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.914941    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9963d70a-6261-48fc-b1b7-aa3138751293-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.914979    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915009    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915053    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915134    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915216    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9963d70a-6261-48fc-b1b7-aa3138751293-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915291    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqk8\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-kube-api-access-wxqk8\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915370 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915360    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915429 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915397    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915751    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915791    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.915972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.915837    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9963d70a-6261-48fc-b1b7-aa3138751293-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.917946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.917925    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.918066 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.917980    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9963d70a-6261-48fc-b1b7-aa3138751293-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.974374 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.974349    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:28.975516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:28.975491    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqk8\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-kube-api-access-wxqk8\") pod \"maas-default-gateway-openshift-default-6886b96f9-2qgzh\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:29.028402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:29.028378    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:29.469155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:29.469127    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"]
Oct 30 20:42:29.469689 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:42:29.469668    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9963d70a_6261_48fc_b1b7_aa3138751293.slice/crio-14a017e804464301d66051a6f6a84ce2eefcfbeab4e55abbdf44f7d17f45a643 WatchSource:0}: Error finding container 14a017e804464301d66051a6f6a84ce2eefcfbeab4e55abbdf44f7d17f45a643: Status 404 returned error can't find the container with id 14a017e804464301d66051a6f6a84ce2eefcfbeab4e55abbdf44f7d17f45a643
Oct 30 20:42:29.472101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:29.472069    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:29.472157 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:29.472134    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:29.472192 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:29.472161    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:30.174763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:30.174729    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" event={"ID":"9963d70a-6261-48fc-b1b7-aa3138751293","Type":"ContainerStarted","Data":"533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498"}
Oct 30 20:42:30.175165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:30.174806    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" event={"ID":"9963d70a-6261-48fc-b1b7-aa3138751293","Type":"ContainerStarted","Data":"14a017e804464301d66051a6f6a84ce2eefcfbeab4e55abbdf44f7d17f45a643"}
Oct 30 20:42:30.353618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:30.353559    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podStartSLOduration=2.35351861 podStartE2EDuration="2.35351861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:30.352308141 +0000 UTC m=+1376.501359128" watchObservedRunningTime="2026-10-30 20:42:30.35351861 +0000 UTC m=+1376.502569597"
Oct 30 20:42:31.028894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.028860    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:31.042735 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.042702    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:42:31.042938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.042903    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:42:31.377942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.377868    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"]
Oct 30 20:42:31.382106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.382089    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.440819 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.440793    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7f0ceebd-serving-cert\""
Oct 30 20:42:31.461657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.461380    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-maas-api-tls\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.461657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.461517    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-metrics-certs\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.461657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.461570    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c667301-cd1f-4c13-896b-056314a0502b-openshift-service-ca\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.461920 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.461705    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7d68\" (UniqueName: \"kubernetes.io/projected/8c667301-cd1f-4c13-896b-056314a0502b-kube-api-access-p7d68\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.499360 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.499333    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"]
Oct 30 20:42:31.563412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.563374    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7d68\" (UniqueName: \"kubernetes.io/projected/8c667301-cd1f-4c13-896b-056314a0502b-kube-api-access-p7d68\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.563580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.563510    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-maas-api-tls\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.563666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.563649    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-metrics-certs\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.563716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.563681    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c667301-cd1f-4c13-896b-056314a0502b-openshift-service-ca\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.564313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.564280    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c667301-cd1f-4c13-896b-056314a0502b-openshift-service-ca\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.566197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.566173    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-maas-api-tls\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.566291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.566228    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-metrics-certs\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.592966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.592910    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:31.653258 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.650231    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7d68\" (UniqueName: \"kubernetes.io/projected/8c667301-cd1f-4c13-896b-056314a0502b-kube-api-access-p7d68\") pod \"maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.696737 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.696702    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:31.785474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:31.784727    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:32.039555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:32.039337    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:42:32.039555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:32.039396    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:42:32.055368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:32.055336    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:32.188955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:32.188915    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"]
Oct 30 20:42:33.031558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:33.028848    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:42:33.031558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:33.028921    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:42:33.199601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:33.199570    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv" event={"ID":"8c667301-cd1f-4c13-896b-056314a0502b","Type":"ContainerStarted","Data":"f40bf53e201dcb60bd5d5198e71f2069a0c663ddd8d83aa2df780400d4677796"}
Oct 30 20:42:33.763662 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:33.763622    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:33.969012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:33.968978    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:34.032618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:34.032177    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:42:34.032618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:34.032232    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:42:34.208060 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:34.208035    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv" event={"ID":"8c667301-cd1f-4c13-896b-056314a0502b","Type":"ContainerStarted","Data":"e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703"}
Oct 30 20:42:34.208255 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:34.208235    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:34.346614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:34.345822    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv" podStartSLOduration=2.379962613 podStartE2EDuration="3.345803975s" totalImagesPullingTime="965.841362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:34.329914327 +0000 UTC m=+1380.478965314" watchObservedRunningTime="2026-10-30 20:42:34.345803975 +0000 UTC m=+1380.494854962"
Oct 30 20:42:35.031562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:35.028859    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:42:35.031562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:35.028914    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:42:36.029161 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:36.029129    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:42:36.029581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:36.029195    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:42:36.364732 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:36.364649    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:36.578335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:36.578296    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:36.781807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:36.781769    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:37.028847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:37.028810    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:42:37.029021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:37.028877    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:42:37.117720 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:42:37.117636    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:42:38.220516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:38.220487    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:38.221241 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:38.221215    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:38.221860 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:38.221843    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:42:40.252744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:40.252715    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:42:42.198109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:42.198022    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:43.816656 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.816618    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"]
Oct 30 20:42:43.822511 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.822492    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.866081 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.866058    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"]
Oct 30 20:42:43.870084 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.870070    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.887815 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.887794    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.887910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.887826    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada2cf2a-29a4-4042-b8be-9536203e8703-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.887951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.887915    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.887985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.887948    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.888021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.887997    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.888021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888014    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.888197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888174    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.888247 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888232    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.888296 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888282    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.888354 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888342    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.888389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888371    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.888428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888390    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddqg6\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-kube-api-access-ddqg6\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.888463 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888440    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zd8\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-kube-api-access-s4zd8\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.888512 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888475    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.888512 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.888489    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.889238 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.889211    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fefae-1b0e-4774-988c-fbe7be25d757-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.889327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.889317    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.889370 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.889341    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.923841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.923817    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"]
Oct 30 20:42:43.990222 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990194    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.990442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990230    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.990442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990259    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.990442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990279    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddqg6\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-kube-api-access-ddqg6\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.990442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990304    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zd8\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-kube-api-access-s4zd8\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.990442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990355    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.990442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990377    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.990442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990390    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fefae-1b0e-4774-988c-fbe7be25d757-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.991212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990453    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.991212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990476    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.991212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990523    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.991212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990578    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada2cf2a-29a4-4042-b8be-9536203e8703-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.991212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990626    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.991212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990661    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.991212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990660    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.992626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.990751    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.992750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.992634    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.992750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.992725    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.992950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.991812    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada2cf2a-29a4-4042-b8be-9536203e8703-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.992950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.992250    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.992950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.992011    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fefae-1b0e-4774-988c-fbe7be25d757-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.992950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.992797    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.993216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.993004    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.993216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.993143    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.993216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.991108    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.993403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.993278    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.993403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.992747    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.993403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.993381    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.996381 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.996356    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.997664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.997639    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.998067 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.998046    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:43.999277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.999250    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:43.999578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:43.999560    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"]
Oct 30 20:42:44.051999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.051970    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddqg6\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-kube-api-access-ddqg6\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:44.052336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.052313    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zd8\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-kube-api-access-s4zd8\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:44.052380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.052341    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xjvz5\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:44.052447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.052429    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-5bb9c\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:44.136069 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.135996    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:44.183402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.183367    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:44.571722 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.571693    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"]
Oct 30 20:42:44.605298 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:42:44.605273    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada2cf2a_29a4_4042_b8be_9536203e8703.slice/crio-190f81571b2092eaa36bb5559ea73c51b9c2f897448422055507e66bd137228e WatchSource:0}: Error finding container 190f81571b2092eaa36bb5559ea73c51b9c2f897448422055507e66bd137228e: Status 404 returned error can't find the container with id 190f81571b2092eaa36bb5559ea73c51b9c2f897448422055507e66bd137228e
Oct 30 20:42:44.607470 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.607437    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:44.607577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.607506    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:44.607577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.607555    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:44.621740 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:42:44.621717    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5fefae_1b0e_4774_988c_fbe7be25d757.slice/crio-f01eb4099491b255a6c427579a354beedfcacb350fa30247f6bb5430cd42c67f WatchSource:0}: Error finding container f01eb4099491b255a6c427579a354beedfcacb350fa30247f6bb5430cd42c67f: Status 404 returned error can't find the container with id f01eb4099491b255a6c427579a354beedfcacb350fa30247f6bb5430cd42c67f
Oct 30 20:42:44.622421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.622403    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"]
Oct 30 20:42:44.624407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.624377    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:44.624500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.624453    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:44.624602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:44.624500    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:45.367463 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:45.367432    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" event={"ID":"ada2cf2a-29a4-4042-b8be-9536203e8703","Type":"ContainerStarted","Data":"03d0ba1750e578350b8971ad09407021e614854bd4a913910be9c2502d77b1a8"}
Oct 30 20:42:45.367934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:45.367481    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" event={"ID":"ada2cf2a-29a4-4042-b8be-9536203e8703","Type":"ContainerStarted","Data":"190f81571b2092eaa36bb5559ea73c51b9c2f897448422055507e66bd137228e"}
Oct 30 20:42:45.369113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:45.369087    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" event={"ID":"de5fefae-1b0e-4774-988c-fbe7be25d757","Type":"ContainerStarted","Data":"fa64e90450ab5c6cad68c33ad20668633fb902bd942814c443e0bf9943eb412d"}
Oct 30 20:42:45.369206 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:45.369129    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" event={"ID":"de5fefae-1b0e-4774-988c-fbe7be25d757","Type":"ContainerStarted","Data":"f01eb4099491b255a6c427579a354beedfcacb350fa30247f6bb5430cd42c67f"}
Oct 30 20:42:45.500274 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:45.500211    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podStartSLOduration=2.500194071 podStartE2EDuration="2.500194071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:45.500127864 +0000 UTC m=+1391.649178852" watchObservedRunningTime="2026-10-30 20:42:45.500194071 +0000 UTC m=+1391.649245058"
Oct 30 20:42:46.136956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:46.136913    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:46.148685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:46.148648    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:42:46.149017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:46.148983    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:42:46.183973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:46.183949    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:46.185087 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:46.185067    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:42:46.185283 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:46.185257    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:42:46.971098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:46.970940    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:47.139430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:47.139396    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:42:47.139739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:47.139708    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:42:47.192954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:47.192914    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:42:47.216673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:47.216626    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:42:47.300191 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:47.299825    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:47.534559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:47.534119    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:48.140559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:48.140140    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:42:48.140559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:48.140211    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:42:48.184002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:48.183967    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:42:48.184165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:48.184031    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:42:49.143922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:49.143878    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:42:49.144419 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:49.143955    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:42:49.184049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:49.184013    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:42:49.184200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:49.184062    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:42:50.138558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:50.136885    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:42:50.138558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:50.136963    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:42:50.184327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:50.184299    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:42:50.184727 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:50.184349    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:42:51.137280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:51.137223    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:42:51.137555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:51.137289    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:42:51.184267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:51.184226    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:42:51.184469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:51.184295    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:42:52.136566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.136512    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:42:52.136773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.136600    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:42:52.184387 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.184355    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:42:52.184574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.184414    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:42:52.251170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.251119    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:52.609100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.609041    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podStartSLOduration=9.609021753 podStartE2EDuration="9.609021753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:45.660860401 +0000 UTC m=+1391.809911391" watchObservedRunningTime="2026-10-30 20:42:52.609021753 +0000 UTC m=+1398.758072740"
Oct 30 20:42:52.615164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.615132    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"]
Oct 30 20:42:52.620890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.620862    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.723609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.723566    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.724004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.723979    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fbjr\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-kube-api-access-6fbjr\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.726735 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.726711    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.727020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.726905    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.727308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.727197    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.727472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.727454    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.727806 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.727785    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.728377 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.728357    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.728749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.728727    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.734578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.734556    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"]
Oct 30 20:42:52.833785 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.833747    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fbjr\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-kube-api-access-6fbjr\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.833943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.833829    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.833943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.833878    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.833943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.833928    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.834435 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.833959    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.834435 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.833990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.834435 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.834121    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.834435 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.834165    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.834435 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.834216    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.835212 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.835182    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.835479 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.835454    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.835763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.835738    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.836388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.836361    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.838293 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.838264    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.866077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.866015    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.866436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.866414    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.921568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.915295    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.924280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.922317    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fbjr\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-kube-api-access-6fbjr\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:52.942554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:52.942329    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:53.442558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.441657    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"]
Oct 30 20:42:53.447955 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:42:53.447910    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a88b3c2_ef05_4597_a222_d46fcf731f9f.slice/crio-4165ddf8e60592314a5fe4aa60735029a61272279aa0f637bb6397319c1c1127 WatchSource:0}: Error finding container 4165ddf8e60592314a5fe4aa60735029a61272279aa0f637bb6397319c1c1127: Status 404 returned error can't find the container with id 4165ddf8e60592314a5fe4aa60735029a61272279aa0f637bb6397319c1c1127
Oct 30 20:42:53.452102 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.452065    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:53.452214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.452149    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:53.452214 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.452195    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:53.471316 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.471287    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:53.473146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.473127    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:53.474466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.474438    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" event={"ID":"4a88b3c2-ef05-4597-a222-d46fcf731f9f","Type":"ContainerStarted","Data":"4165ddf8e60592314a5fe4aa60735029a61272279aa0f637bb6397319c1c1127"}
Oct 30 20:42:53.486598 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.480952    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:42:53.522515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.522489    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:53.523640 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.523620    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:53.524293 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:53.524272    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:42:54.482516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:54.482474    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" event={"ID":"4a88b3c2-ef05-4597-a222-d46fcf731f9f","Type":"ContainerStarted","Data":"76194b177946df34f4fe6387f313300d46aa5239b4e16de80c68c86b3feb6fcb"}
Oct 30 20:42:54.603376 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:54.603331    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podStartSLOduration=2.603318275 podStartE2EDuration="2.603318275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:54.601911619 +0000 UTC m=+1400.750962668" watchObservedRunningTime="2026-10-30 20:42:54.603318275 +0000 UTC m=+1400.752369261"
Oct 30 20:42:54.945319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:54.945226    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:42:54.947287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:54.947259    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:42:54.947440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:54.947316    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:42:55.943594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:55.943522    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:42:55.944005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:55.943626    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:42:56.944558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:56.943762    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:42:56.944558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:56.943830    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:42:57.338571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:57.333605    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:57.944563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:57.942977    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:42:57.944563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:57.943030    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:42:58.943560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:58.943099    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:42:58.943560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:58.943152    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:42:59.943217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:59.943178    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:42:59.943844 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:42:59.943235    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:43:01.294968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:01.294929    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:43:01.295466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:01.295369    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:43:01.296222 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:01.296201    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:43:02.125931 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:02.125885    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:04.075566 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:04.075522    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:43:06.985517 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:06.985476    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:07.142020 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:43:07.141981    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:43:13.634720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:13.634670    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:13.980671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:13.980644    5002 patch_prober.go:29] interesting pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:43:13.980836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:13.980695    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:43:15.701671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:15.701640    5002 generic.go:356] "Generic (PLEG): container finished" podID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerID="42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381" exitCode=0
Oct 30 20:43:15.702088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:15.701719    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" event={"ID":"46ffc7a3-1190-4dd1-9723-c7821df2fba4","Type":"ContainerDied","Data":"42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381"}
Oct 30 20:43:15.702136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:15.702095    5002 scope.go:122] "RemoveContainer" containerID="42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381"
Oct 30 20:43:15.703456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:15.703424    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:15.703599 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:15.703507    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:15.703599 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:15.703576    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:16.385347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:16.385312    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:16.734111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:16.734069    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" event={"ID":"46ffc7a3-1190-4dd1-9723-c7821df2fba4","Type":"ContainerStarted","Data":"41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d"}
Oct 30 20:43:16.946948 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:16.946914    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:43:16.947136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:16.947119    5002 patch_prober.go:29] interesting pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:43:16.947190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:16.947170    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 20:43:17.947824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:17.947793    5002 patch_prober.go:29] interesting pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:43:17.948232 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:17.947847    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 20:43:18.947961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:18.947927    5002 patch_prober.go:29] interesting pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:43:18.948354 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:18.947986    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 20:43:19.946846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:19.946814    5002 patch_prober.go:29] interesting pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:43:19.947018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:19.946860    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 20:43:20.947371 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:20.947337    5002 patch_prober.go:29] interesting pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:43:20.947816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:20.947384    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 20:43:21.164408 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:21.163736    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:21.951181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:21.951154    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:43:21.951690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:21.951649    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:43:21.952276 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:21.952249    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:43:22.186252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:22.186215    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:25.838018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:25.837976    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:28.456483 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:28.456453    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:28.689458 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:28.689432    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:30.357396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:30.357358    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:32.180589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:32.180556    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:35.527191 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:35.527153    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:37.116426 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:43:37.116383    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:43:39.366935 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:39.366903    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:40.611397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:40.611353    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:45.710591 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:45.710277    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:47.389293 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:47.389260    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"]
Oct 30 20:43:47.504450 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:47.504414    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"]
Oct 30 20:43:47.799766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:47.799729    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:47.982710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:47.982673    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:50.092781 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:50.092748    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"]
Oct 30 20:43:50.495187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:50.495153    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:50.677119 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:50.677077    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:51.095259 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:51.095225    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:52.389802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.389720    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz" podUID="fe1bf291-d004-4bd8-b8bd-438959bf29de" containerName="payload-processing" containerID="cri-o://5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7" gracePeriod=25
Oct 30 20:43:52.505002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.504927    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg" podUID="26ec05a9-b960-4528-a9f0-2133f5a6ae00" containerName="payload-pre-processing" containerID="cri-o://0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b" gracePeriod=25
Oct 30 20:43:52.696953 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.696853    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:43:52.758218 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.758187    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1bf291-d004-4bd8-b8bd-438959bf29de-kube-api-access-xwbxb\" (UniqueName: \"kubernetes.io/projected/fe1bf291-d004-4bd8-b8bd-438959bf29de-kube-api-access-xwbxb\") pod \"fe1bf291-d004-4bd8-b8bd-438959bf29de\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") "
Oct 30 20:43:52.758396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.758244    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1bf291-d004-4bd8-b8bd-438959bf29de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1bf291-d004-4bd8-b8bd-438959bf29de-plugins-config-volume\") pod \"fe1bf291-d004-4bd8-b8bd-438959bf29de\" (UID: \"fe1bf291-d004-4bd8-b8bd-438959bf29de\") "
Oct 30 20:43:52.758922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.758894    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1bf291-d004-4bd8-b8bd-438959bf29de-plugins-config-volume" pod "fe1bf291-d004-4bd8-b8bd-438959bf29de" (UID: "fe1bf291-d004-4bd8-b8bd-438959bf29de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:52.762150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.762123    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1bf291-d004-4bd8-b8bd-438959bf29de-kube-api-access-xwbxb" pod "fe1bf291-d004-4bd8-b8bd-438959bf29de" (UID: "fe1bf291-d004-4bd8-b8bd-438959bf29de"). InnerVolumeSpecName "kube-api-access-xwbxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:52.792798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.792779    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:43:52.860369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.860338    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ec05a9-b960-4528-a9f0-2133f5a6ae00-kube-api-access-2fzt7\" (UniqueName: \"kubernetes.io/projected/26ec05a9-b960-4528-a9f0-2133f5a6ae00-kube-api-access-2fzt7\") pod \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") "
Oct 30 20:43:52.860938 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.860390    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ec05a9-b960-4528-a9f0-2133f5a6ae00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec05a9-b960-4528-a9f0-2133f5a6ae00-plugins-config-volume\") pod \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\" (UID: \"26ec05a9-b960-4528-a9f0-2133f5a6ae00\") "
Oct 30 20:43:52.861120 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.861092    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwbxb\" (UniqueName: \"kubernetes.io/projected/fe1bf291-d004-4bd8-b8bd-438959bf29de-kube-api-access-xwbxb\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:52.861197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.861121    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1bf291-d004-4bd8-b8bd-438959bf29de-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:52.861465 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.861442    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ec05a9-b960-4528-a9f0-2133f5a6ae00-plugins-config-volume" pod "26ec05a9-b960-4528-a9f0-2133f5a6ae00" (UID: "26ec05a9-b960-4528-a9f0-2133f5a6ae00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:52.864461 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.864429    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ec05a9-b960-4528-a9f0-2133f5a6ae00-kube-api-access-2fzt7" pod "26ec05a9-b960-4528-a9f0-2133f5a6ae00" (UID: "26ec05a9-b960-4528-a9f0-2133f5a6ae00"). InnerVolumeSpecName "kube-api-access-2fzt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:52.962770 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.962738    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzt7\" (UniqueName: \"kubernetes.io/projected/26ec05a9-b960-4528-a9f0-2133f5a6ae00-kube-api-access-2fzt7\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:52.962911 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:52.962775    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec05a9-b960-4528-a9f0-2133f5a6ae00-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:53.076234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.076199    5002 generic.go:356] "Generic (PLEG): container finished" podID="26ec05a9-b960-4528-a9f0-2133f5a6ae00" containerID="0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b" exitCode=0
Oct 30 20:43:53.076406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.076245    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"
Oct 30 20:43:53.076406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.076287    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg" event={"ID":"26ec05a9-b960-4528-a9f0-2133f5a6ae00","Type":"ContainerDied","Data":"0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b"}
Oct 30 20:43:53.076406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.076338    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg" event={"ID":"26ec05a9-b960-4528-a9f0-2133f5a6ae00","Type":"ContainerDied","Data":"1146aa9502eff8c39c16ceb1ca95d1ea7ef356b214c23980c617ced11c4204f4"}
Oct 30 20:43:53.076406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.076360    5002 scope.go:122] "RemoveContainer" containerID="0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b"
Oct 30 20:43:53.078033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.077954    5002 generic.go:356] "Generic (PLEG): container finished" podID="fe1bf291-d004-4bd8-b8bd-438959bf29de" containerID="5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7" exitCode=0
Oct 30 20:43:53.078033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.077998    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"
Oct 30 20:43:53.078033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.078002    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz" event={"ID":"fe1bf291-d004-4bd8-b8bd-438959bf29de","Type":"ContainerDied","Data":"5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7"}
Oct 30 20:43:53.078271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.078054    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz" event={"ID":"fe1bf291-d004-4bd8-b8bd-438959bf29de","Type":"ContainerDied","Data":"c303df5b05a060cbb0e2dcf4e5747ff349457fa273648e4305fcaa2128b22605"}
Oct 30 20:43:53.089094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.089079    5002 scope.go:122] "RemoveContainer" containerID="0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b"
Oct 30 20:43:53.089360 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:43:53.089341    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b\": container with ID starting with 0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b not found: ID does not exist" containerID="0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b"
Oct 30 20:43:53.089406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.089366    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b"} err="failed to get container status \"0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b\": rpc error: code = NotFound desc = could not find container \"0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b\": container with ID starting with 0ea50d881b3041cc164c71e1d84db05e620296a4b405c0f05f0ff7a9cb368a7b not found: ID does not exist"
Oct 30 20:43:53.089406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.089381    5002 scope.go:122] "RemoveContainer" containerID="5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7"
Oct 30 20:43:53.100654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.100640    5002 scope.go:122] "RemoveContainer" containerID="5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7"
Oct 30 20:43:53.100906 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:43:53.100890    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7\": container with ID starting with 5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7 not found: ID does not exist" containerID="5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7"
Oct 30 20:43:53.100969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.100912    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7"} err="failed to get container status \"5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7\": rpc error: code = NotFound desc = could not find container \"5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7\": container with ID starting with 5abb0608dd639e5287c15900b5ab000e0b20fc709ccc8e6cbb7297be45b14ef7 not found: ID does not exist"
Oct 30 20:43:53.244302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.244241    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"]
Oct 30 20:43:53.248018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.247997    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz"]
Oct 30 20:43:53.358041 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.358012    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"]
Oct 30 20:43:53.364317 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.364296    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg"]
Oct 30 20:43:53.969071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.969035    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ec05a9-b960-4528-a9f0-2133f5a6ae00" path="/var/lib/kubelet/pods/26ec05a9-b960-4528-a9f0-2133f5a6ae00/volumes"
Oct 30 20:43:53.969728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:53.969709    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1bf291-d004-4bd8-b8bd-438959bf29de" path="/var/lib/kubelet/pods/fe1bf291-d004-4bd8-b8bd-438959bf29de/volumes"
Oct 30 20:43:55.093006 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.092966    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn" podUID="2deb2c54-6bd4-4a9f-843e-c799db781430" containerName="payload-processing" containerID="cri-o://db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b" gracePeriod=25
Oct 30 20:43:55.132170 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:43:55.132153    5002 logging.go:55] [core] [Channel #1235 SubChannel #1236]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.92:50052", ServerName: "10.132.0.92:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.92:50052: connect: connection refused"
Oct 30 20:43:55.400408 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.400382    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:43:55.501703 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.501675    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2deb2c54-6bd4-4a9f-843e-c799db781430-kube-api-access-n7c7p\" (UniqueName: \"kubernetes.io/projected/2deb2c54-6bd4-4a9f-843e-c799db781430-kube-api-access-n7c7p\") pod \"2deb2c54-6bd4-4a9f-843e-c799db781430\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") "
Oct 30 20:43:55.501878 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.501764    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2deb2c54-6bd4-4a9f-843e-c799db781430-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2deb2c54-6bd4-4a9f-843e-c799db781430-plugins-config-volume\") pod \"2deb2c54-6bd4-4a9f-843e-c799db781430\" (UID: \"2deb2c54-6bd4-4a9f-843e-c799db781430\") "
Oct 30 20:43:55.502210 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.502170    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deb2c54-6bd4-4a9f-843e-c799db781430-plugins-config-volume" pod "2deb2c54-6bd4-4a9f-843e-c799db781430" (UID: "2deb2c54-6bd4-4a9f-843e-c799db781430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:55.503999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.503980    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deb2c54-6bd4-4a9f-843e-c799db781430-kube-api-access-n7c7p" pod "2deb2c54-6bd4-4a9f-843e-c799db781430" (UID: "2deb2c54-6bd4-4a9f-843e-c799db781430"). InnerVolumeSpecName "kube-api-access-n7c7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:55.603760 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.603730    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7c7p\" (UniqueName: \"kubernetes.io/projected/2deb2c54-6bd4-4a9f-843e-c799db781430-kube-api-access-n7c7p\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:55.603760 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:55.603755    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2deb2c54-6bd4-4a9f-843e-c799db781430-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:56.098468 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.098436    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:56.118847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.112523    5002 generic.go:356] "Generic (PLEG): container finished" podID="2deb2c54-6bd4-4a9f-843e-c799db781430" containerID="db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b" exitCode=0
Oct 30 20:43:56.118847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.112571    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn" event={"ID":"2deb2c54-6bd4-4a9f-843e-c799db781430","Type":"ContainerDied","Data":"db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b"}
Oct 30 20:43:56.118847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.112617    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"
Oct 30 20:43:56.118847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.112642    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn" event={"ID":"2deb2c54-6bd4-4a9f-843e-c799db781430","Type":"ContainerDied","Data":"1a55b729f9f48c84068f2dcf29ecb3576a3fd1f13f739f920ba9bfb9a694ab28"}
Oct 30 20:43:56.118847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.112670    5002 scope.go:122] "RemoveContainer" containerID="db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b"
Oct 30 20:43:56.132601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.132509    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn" podUID="2deb2c54-6bd4-4a9f-843e-c799db781430" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.92:50052\" within 1s: context deadline exceeded"
Oct 30 20:43:56.132899 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:43:56.132562    5002 logging.go:55] [core] [Channel #1235 SubChannel #1236]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.92:50052", ServerName: "10.132.0.92:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.92:50052: operation was canceled"
Oct 30 20:43:56.137267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.137242    5002 scope.go:122] "RemoveContainer" containerID="db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b"
Oct 30 20:43:56.137666 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:43:56.137639    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b\": container with ID starting with db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b not found: ID does not exist" containerID="db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b"
Oct 30 20:43:56.137775 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.137675    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b"} err="failed to get container status \"db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b\": rpc error: code = NotFound desc = could not find container \"db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b\": container with ID starting with db62356a42c2ae920eed639c5c7fe82d0da9c947626561c49bda94bcfe837f4b not found: ID does not exist"
Oct 30 20:43:56.280676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.280589    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"]
Oct 30 20:43:56.284284 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:56.284258    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn"]
Oct 30 20:43:57.447101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:57.447063    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:57.968742 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:43:57.968713    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2deb2c54-6bd4-4a9f-843e-c799db781430" path="/var/lib/kubelet/pods/2deb2c54-6bd4-4a9f-843e-c799db781430/volumes"
Oct 30 20:44:00.334554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:00.332782    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:00.527160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:00.527121    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:01.000438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:01.000406    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:02.635111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:02.635072    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:05.783206 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:05.783174    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:07.115783 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:44:07.115744    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:44:08.517597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:08.517561    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:10.509565 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:10.509057    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:15.318092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:15.317998    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:19.262670 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.262637    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"]
Oct 30 20:44:19.263184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.262827    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv" podUID="8c667301-cd1f-4c13-896b-056314a0502b" containerName="maas-api" containerID="cri-o://e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703" gracePeriod=30
Oct 30 20:44:19.404583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.404549    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:19.602187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.602156    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:44:19.646912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.646875    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c667301-cd1f-4c13-896b-056314a0502b-kube-api-access-p7d68\" (UniqueName: \"kubernetes.io/projected/8c667301-cd1f-4c13-896b-056314a0502b-kube-api-access-p7d68\") pod \"8c667301-cd1f-4c13-896b-056314a0502b\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") "
Oct 30 20:44:19.647114 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.647022    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-metrics-certs\") pod \"8c667301-cd1f-4c13-896b-056314a0502b\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") "
Oct 30 20:44:19.647114 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.647068    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c667301-cd1f-4c13-896b-056314a0502b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c667301-cd1f-4c13-896b-056314a0502b-openshift-service-ca\") pod \"8c667301-cd1f-4c13-896b-056314a0502b\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") "
Oct 30 20:44:19.647417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.647175    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-maas-api-tls\") pod \"8c667301-cd1f-4c13-896b-056314a0502b\" (UID: \"8c667301-cd1f-4c13-896b-056314a0502b\") "
Oct 30 20:44:19.649380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.649354    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c667301-cd1f-4c13-896b-056314a0502b-openshift-service-ca" pod "8c667301-cd1f-4c13-896b-056314a0502b" (UID: "8c667301-cd1f-4c13-896b-056314a0502b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:19.661689 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.661658    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-maas-api-tls" pod "8c667301-cd1f-4c13-896b-056314a0502b" (UID: "8c667301-cd1f-4c13-896b-056314a0502b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:19.669756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.669721    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c667301-cd1f-4c13-896b-056314a0502b-kube-api-access-p7d68" pod "8c667301-cd1f-4c13-896b-056314a0502b" (UID: "8c667301-cd1f-4c13-896b-056314a0502b"). InnerVolumeSpecName "kube-api-access-p7d68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:19.671835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.671807    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-metrics-certs" pod "8c667301-cd1f-4c13-896b-056314a0502b" (UID: "8c667301-cd1f-4c13-896b-056314a0502b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:19.749843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.749805    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c667301-cd1f-4c13-896b-056314a0502b-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:19.749843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.749844    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:19.750101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.749861    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7d68\" (UniqueName: \"kubernetes.io/projected/8c667301-cd1f-4c13-896b-056314a0502b-kube-api-access-p7d68\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:19.750101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:19.749875    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c667301-cd1f-4c13-896b-056314a0502b-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:20.299981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.299890    5002 generic.go:356] "Generic (PLEG): container finished" podID="8c667301-cd1f-4c13-896b-056314a0502b" containerID="e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703" exitCode=0
Oct 30 20:44:20.299981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.299958    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"
Oct 30 20:44:20.299981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.299968    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv" event={"ID":"8c667301-cd1f-4c13-896b-056314a0502b","Type":"ContainerDied","Data":"e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703"}
Oct 30 20:44:20.300492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.300056    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv" event={"ID":"8c667301-cd1f-4c13-896b-056314a0502b","Type":"ContainerDied","Data":"f40bf53e201dcb60bd5d5198e71f2069a0c663ddd8d83aa2df780400d4677796"}
Oct 30 20:44:20.300492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.300094    5002 scope.go:122] "RemoveContainer" containerID="e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703"
Oct 30 20:44:20.317098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.317083    5002 scope.go:122] "RemoveContainer" containerID="e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703"
Oct 30 20:44:20.317579 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:44:20.317558    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703\": container with ID starting with e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703 not found: ID does not exist" containerID="e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703"
Oct 30 20:44:20.317696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.317586    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703"} err="failed to get container status \"e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703\": rpc error: code = NotFound desc = could not find container \"e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703\": container with ID starting with e580ff8fab15162d9955206cb48650336b78ae78efce3c0ab1118297285c1703 not found: ID does not exist"
Oct 30 20:44:20.471851 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.471819    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"]
Oct 30 20:44:20.478042 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.478013    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7f0ceebd-54c65d69bb-jl2kv"]
Oct 30 20:44:20.534519 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.534486    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:20.913323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:20.913284    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:21.258393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:21.258355    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:21.730580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:21.730314    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"]
Oct 30 20:44:21.730580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:21.730548    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" containerID="cri-o://41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d" gracePeriod=30
Oct 30 20:44:21.774630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:21.774598    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:21.984154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:21.979836    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c667301-cd1f-4c13-896b-056314a0502b" path="/var/lib/kubelet/pods/8c667301-cd1f-4c13-896b-056314a0502b/volumes"
Oct 30 20:44:21.984154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:21.980094    5002 patch_prober.go:29] interesting pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:44:21.984154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:21.980148    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:44:22.406462 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.406382    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:22.588916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.588887    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:22.775777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.775746    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:22.969266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.969232    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:22.975717 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.975687    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"]
Oct 30 20:44:22.978177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.978151    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fe1bf291-d004-4bd8-b8bd-438959bf29de" containerName="payload-processing"
Oct 30 20:44:22.978354 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.978340    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1bf291-d004-4bd8-b8bd-438959bf29de" containerName="payload-processing"
Oct 30 20:44:22.978484 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.978472    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2deb2c54-6bd4-4a9f-843e-c799db781430" containerName="payload-processing"
Oct 30 20:44:22.978650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.978633    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2deb2c54-6bd4-4a9f-843e-c799db781430" containerName="payload-processing"
Oct 30 20:44:22.978837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.978826    5002 cpu_manager.go:414] "containerMap: removing container" podUID="26ec05a9-b960-4528-a9f0-2133f5a6ae00" containerName="payload-pre-processing"
Oct 30 20:44:22.978923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.978912    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ec05a9-b960-4528-a9f0-2133f5a6ae00" containerName="payload-pre-processing"
Oct 30 20:44:22.979012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.979002    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8c667301-cd1f-4c13-896b-056314a0502b" containerName="maas-api"
Oct 30 20:44:22.979092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.979082    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c667301-cd1f-4c13-896b-056314a0502b" containerName="maas-api"
Oct 30 20:44:22.979659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.979641    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2deb2c54-6bd4-4a9f-843e-c799db781430" containerName="payload-processing"
Oct 30 20:44:22.979811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.979788    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ec05a9-b960-4528-a9f0-2133f5a6ae00" containerName="payload-pre-processing"
Oct 30 20:44:22.979902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.979890    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c667301-cd1f-4c13-896b-056314a0502b" containerName="maas-api"
Oct 30 20:44:22.979988 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.979978    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1bf291-d004-4bd8-b8bd-438959bf29de" containerName="payload-processing"
Oct 30 20:44:22.994555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:22.984547    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018212    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-token\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018279    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-data\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018369    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-socket\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018403    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-credential-socket\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018440    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-envoy\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018487    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8db72a9-39e5-4274-aff2-856577cfe58d-istiod-ca-cert\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018617    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967wz\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-kube-api-access-967wz\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018665    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-podinfo\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.021554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.018702    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-certs\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.067556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.058611    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"]
Oct 30 20:44:23.092927 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.092887    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0d757729-openshift-default-dockercfg-zwv66\""
Oct 30 20:44:23.133559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.126773    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-socket\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.165737 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.165707    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-credential-socket\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.165952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.165938    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-envoy\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.166043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.166034    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8db72a9-39e5-4274-aff2-856577cfe58d-istiod-ca-cert\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.166186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.166176    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967wz\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-kube-api-access-967wz\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.166247 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.166239    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-podinfo\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.166325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.166313    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-certs\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.166621 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.166605    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-token\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.166744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.166733    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-data\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.167272 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.167254    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-data\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.167392 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.134175    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-socket\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.167714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.167699    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-credential-socket\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.169287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.169011    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8db72a9-39e5-4274-aff2-856577cfe58d-istiod-ca-cert\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.172545 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.171798    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-certs\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.173582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.172759    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-envoy\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.173582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.173219    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-podinfo\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.221727 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.221695    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:23.240588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.232203    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-967wz\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-kube-api-access-967wz\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.243453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.243038    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-token\") pod \"e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") " pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.349814 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.349706    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:23.546888 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.546853    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:23.801484 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.801456    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"]
Oct 30 20:44:23.803310 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:44:23.803283    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8db72a9_39e5_4274_aff2_856577cfe58d.slice/crio-60316900c515c4dbe23dcc4c3fa56159dce21e8720529a447f1aa16c6e15ee50 WatchSource:0}: Error finding container 60316900c515c4dbe23dcc4c3fa56159dce21e8720529a447f1aa16c6e15ee50: Status 404 returned error can't find the container with id 60316900c515c4dbe23dcc4c3fa56159dce21e8720529a447f1aa16c6e15ee50
Oct 30 20:44:23.806467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.806420    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:23.806680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.806518    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:23.806680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:23.806592    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:24.072101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:24.072005    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:24.257088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:24.257053    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:24.341710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:24.341639    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" event={"ID":"e8db72a9-39e5-4274-aff2-856577cfe58d","Type":"ContainerStarted","Data":"85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849"}
Oct 30 20:44:24.341850 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:24.341719    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" event={"ID":"e8db72a9-39e5-4274-aff2-856577cfe58d","Type":"ContainerStarted","Data":"60316900c515c4dbe23dcc4c3fa56159dce21e8720529a447f1aa16c6e15ee50"}
Oct 30 20:44:24.351109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:24.351084    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:24.356691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:24.356670    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:24.469331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:24.469285    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podStartSLOduration=2.469271523 podStartE2EDuration="2.469271523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:24.465776959 +0000 UTC m=+1490.614827945" watchObservedRunningTime="2026-10-30 20:44:24.469271523 +0000 UTC m=+1490.618322510"
Oct 30 20:44:25.348629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.348577    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:25.349476 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.349458    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:44:25.366451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.366427    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"]
Oct 30 20:44:25.370779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.370760    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.449447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.449413    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"]
Oct 30 20:44:25.463455 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.463432    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c2bdb58a-openshift-default-dockercfg-7j6gq\""
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.509523    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.510496    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvsc\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-kube-api-access-dnvsc\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.510559    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-credential-socket\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.510708    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-socket\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.510847    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-podinfo\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.510890    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istiod-ca-cert\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.511007    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-token\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.511086    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-certs\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.511192    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-data\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.511552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.511316    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-envoy\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612203    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-socket\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612269    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-podinfo\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612302    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istiod-ca-cert\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612342    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-token\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612412    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-certs\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612521    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-data\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612669    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-envoy\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612742    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvsc\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-kube-api-access-dnvsc\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612778    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-credential-socket\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612891 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612668    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-socket\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.612945 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612897    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-certs\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.613003 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.612970    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-data\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.613068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.613048    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istiod-ca-cert\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.613136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.613118    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-credential-socket\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.615058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.615035    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-envoy\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.615228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.615211    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-podinfo\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.677626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.677592    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-token\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.693642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.691803    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvsc\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-kube-api-access-dnvsc\") pod \"e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") " pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:25.758587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.757027    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:25.984682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:25.984652    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:26.416841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:26.416771    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"]
Oct 30 20:44:26.451885 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:44:26.451849    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88c8a7e_8ab5_466b_abfb_2ca8de7a914d.slice/crio-6d39b0264a0f3e709c76edd75297ecc01861440b07a9eae9e308fa1c4be63867 WatchSource:0}: Error finding container 6d39b0264a0f3e709c76edd75297ecc01861440b07a9eae9e308fa1c4be63867: Status 404 returned error can't find the container with id 6d39b0264a0f3e709c76edd75297ecc01861440b07a9eae9e308fa1c4be63867
Oct 30 20:44:26.454250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:26.454219    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:26.454335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:26.454281    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:26.454335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:26.454312    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:26.724358 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:26.724332    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:27.034429 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.034404    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:44:27.133069 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133037    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istiod-ca-cert\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133091    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-envoy\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133111    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-socket\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133224    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-kube-api-access-c5xvg\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-kube-api-access-c5xvg\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133284    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-podinfo\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133350    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-data\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133385    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-socket" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.133428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133417    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istiod-ca-cert" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:27.133618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133493    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-certs\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133588    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-credential-socket\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133721 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133667    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-token\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-token\") pod \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\" (UID: \"46ffc7a3-1190-4dd1-9723-c7821df2fba4\") "
Oct 30 20:44:27.133963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133768    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-certs" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.133963 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.133835    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-data" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.134185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.134139    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-credential-socket" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.134252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.134224    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.134252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.134244    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.134347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.134260    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.134347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.134272    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.135499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.135474    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-kube-api-access-c5xvg" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "kube-api-access-c5xvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:27.135663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.135643    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-envoy" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.136010 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.135986    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-podinfo" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:44:27.136200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.136185    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-token" pod "46ffc7a3-1190-4dd1-9723-c7821df2fba4" (UID: "46ffc7a3-1190-4dd1-9723-c7821df2fba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:27.236369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.236333    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5xvg\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-kube-api-access-c5xvg\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.236369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.236373    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.236620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.236388    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.236620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.236403    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.236620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.236416    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ffc7a3-1190-4dd1-9723-c7821df2fba4-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.364986 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.364900    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" event={"ID":"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d","Type":"ContainerStarted","Data":"9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564"}
Oct 30 20:44:27.365141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.364993    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" event={"ID":"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d","Type":"ContainerStarted","Data":"6d39b0264a0f3e709c76edd75297ecc01861440b07a9eae9e308fa1c4be63867"}
Oct 30 20:44:27.366691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.366663    5002 generic.go:356] "Generic (PLEG): container finished" podID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerID="41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d" exitCode=0
Oct 30 20:44:27.366816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.366727    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"
Oct 30 20:44:27.366816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.366746    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" event={"ID":"46ffc7a3-1190-4dd1-9723-c7821df2fba4","Type":"ContainerDied","Data":"41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d"}
Oct 30 20:44:27.366816 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.366797    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh" event={"ID":"46ffc7a3-1190-4dd1-9723-c7821df2fba4","Type":"ContainerDied","Data":"fdb8d465fe6b24ea62dbbb35adcfaa7daa5801edcf2addcfa4771a16c046e4ad"}
Oct 30 20:44:27.366978 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.366827    5002 scope.go:122] "RemoveContainer" containerID="41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d"
Oct 30 20:44:27.378046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.378028    5002 scope.go:122] "RemoveContainer" containerID="42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381"
Oct 30 20:44:27.387135 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.387110    5002 scope.go:122] "RemoveContainer" containerID="41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d"
Oct 30 20:44:27.387394 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:44:27.387373    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d\": container with ID starting with 41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d not found: ID does not exist" containerID="41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d"
Oct 30 20:44:27.387454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.387402    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d"} err="failed to get container status \"41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d\": rpc error: code = NotFound desc = could not find container \"41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d\": container with ID starting with 41104462522d38956b5e3953cae2a891bccfa0c67908c41997f321f4dccde54d not found: ID does not exist"
Oct 30 20:44:27.387454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.387419    5002 scope.go:122] "RemoveContainer" containerID="42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381"
Oct 30 20:44:27.387661 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:44:27.387644    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381\": container with ID starting with 42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381 not found: ID does not exist" containerID="42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381"
Oct 30 20:44:27.387712 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.387666    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381"} err="failed to get container status \"42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381\": rpc error: code = NotFound desc = could not find container \"42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381\": container with ID starting with 42ea3775d8939734144bc606195b7cc15110d1d9523dd1e6f7d687ee51b22381 not found: ID does not exist"
Oct 30 20:44:27.498931 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.498885    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" podStartSLOduration=2.498873332 podStartE2EDuration="2.498873332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:27.495420723 +0000 UTC m=+1493.644471709" watchObservedRunningTime="2026-10-30 20:44:27.498873332 +0000 UTC m=+1493.647924319"
Oct 30 20:44:27.617273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.617198    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"]
Oct 30 20:44:27.623802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.623774    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh"]
Oct 30 20:44:27.995921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.995869    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" path="/var/lib/kubelet/pods/46ffc7a3-1190-4dd1-9723-c7821df2fba4/volumes"
Oct 30 20:44:27.996425 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:27.996399    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:28.007685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:28.006296    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:28.375381 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:28.375313    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:28.376219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:28.376202    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:44:28.893939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:28.893906    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:29.643581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:29.643545    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:29.835370 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:29.835340    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:30.044557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.042722    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:30.241962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.241920    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:30.335182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.335105    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"]
Oct 30 20:44:30.335758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.335742    5002 cpu_manager.go:414] "containerMap: removing container" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy"
Oct 30 20:44:30.335758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.335757    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy"
Oct 30 20:44:30.335931 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.335772    5002 cpu_manager.go:414] "containerMap: removing container" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy"
Oct 30 20:44:30.335931 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.335777    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy"
Oct 30 20:44:30.335931 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.335897    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy"
Oct 30 20:44:30.335931 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.335907    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ffc7a3-1190-4dd1-9723-c7821df2fba4" containerName="istio-proxy"
Oct 30 20:44:30.340791 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.340772    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.397972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.397945    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"]
Oct 30 20:44:30.438420 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.438384    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2z9m\" (UniqueName: \"kubernetes.io/projected/e094724d-5b57-4ff1-a451-05be1852311c-kube-api-access-s2z9m\") pod \"payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.439008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.438984    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e094724d-5b57-4ff1-a451-05be1852311c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.460001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.451790    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0d757729\""
Oct 30 20:44:30.460001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.451920    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0d757729-dockercfg-c95nw\""
Oct 30 20:44:30.536568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.530206    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:30.618427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.618330    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2z9m\" (UniqueName: \"kubernetes.io/projected/e094724d-5b57-4ff1-a451-05be1852311c-kube-api-access-s2z9m\") pod \"payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.621449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.621420    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e094724d-5b57-4ff1-a451-05be1852311c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.622439 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.622395    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e094724d-5b57-4ff1-a451-05be1852311c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.627548 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.627031    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"]
Oct 30 20:44:30.662083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.659952    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:30.664950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.664923    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"]
Oct 30 20:44:30.678577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.676110    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2z9m\" (UniqueName: \"kubernetes.io/projected/e094724d-5b57-4ff1-a451-05be1852311c-kube-api-access-s2z9m\") pod \"payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.824710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.824675    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9wt\" (UniqueName: \"kubernetes.io/projected/676ff44f-01bf-428f-a784-d829bec9499b-kube-api-access-xq9wt\") pod \"payload-processing-e2e-ait-0d757729-7857666684-6pxgx\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") " pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:30.824866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.824784    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ff44f-01bf-428f-a784-d829bec9499b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d757729-7857666684-6pxgx\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") " pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:30.926827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.926756    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9wt\" (UniqueName: \"kubernetes.io/projected/676ff44f-01bf-428f-a784-d829bec9499b-kube-api-access-xq9wt\") pod \"payload-processing-e2e-ait-0d757729-7857666684-6pxgx\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") " pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:30.926827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.926809    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ff44f-01bf-428f-a784-d829bec9499b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d757729-7857666684-6pxgx\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") " pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:30.927651 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.927631    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ff44f-01bf-428f-a784-d829bec9499b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d757729-7857666684-6pxgx\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") " pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:30.956456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.956429    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:30.987773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:30.987744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9wt\" (UniqueName: \"kubernetes.io/projected/676ff44f-01bf-428f-a784-d829bec9499b-kube-api-access-xq9wt\") pod \"payload-processing-e2e-ait-0d757729-7857666684-6pxgx\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") " pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:31.280791 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:31.280752    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:31.355339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:31.355305    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"]
Oct 30 20:44:31.403339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:31.403259    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7" event={"ID":"e094724d-5b57-4ff1-a451-05be1852311c","Type":"ContainerStarted","Data":"74c31dab32ecda5ce4cc40f84f985d404f2ff253bc863ea30a8bf58484fe7f4f"}
Oct 30 20:44:31.660841 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:44:31.660810    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676ff44f_01bf_428f_a784_d829bec9499b.slice/crio-1e145959b654042800210fba58dd5a113c06d4016f81d2fc47e24463354abc9b WatchSource:0}: Error finding container 1e145959b654042800210fba58dd5a113c06d4016f81d2fc47e24463354abc9b: Status 404 returned error can't find the container with id 1e145959b654042800210fba58dd5a113c06d4016f81d2fc47e24463354abc9b
Oct 30 20:44:31.661136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:31.661112    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"]
Oct 30 20:44:31.762005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:31.761951    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:31.966085 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:31.966061    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:32.413514 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.413417    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx" event={"ID":"676ff44f-01bf-428f-a784-d829bec9499b","Type":"ContainerStarted","Data":"550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b"}
Oct 30 20:44:32.413514 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.413480    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx" event={"ID":"676ff44f-01bf-428f-a784-d829bec9499b","Type":"ContainerStarted","Data":"1e145959b654042800210fba58dd5a113c06d4016f81d2fc47e24463354abc9b"}
Oct 30 20:44:32.414027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.413526    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:32.415852 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.415822    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7" event={"ID":"e094724d-5b57-4ff1-a451-05be1852311c","Type":"ContainerStarted","Data":"e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c"}
Oct 30 20:44:32.415980 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.415949    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:32.520194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.520163    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:32.534578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.534516    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx" podStartSLOduration=2.53450543 podStartE2EDuration="2.53450543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:32.530557268 +0000 UTC m=+1498.679608256" watchObservedRunningTime="2026-10-30 20:44:32.53450543 +0000 UTC m=+1498.683556416"
Oct 30 20:44:32.653904 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:32.653838    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7" podStartSLOduration=2.653818008 podStartE2EDuration="2.653818008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:32.651848008 +0000 UTC m=+1498.800898995" watchObservedRunningTime="2026-10-30 20:44:32.653818008 +0000 UTC m=+1498.802868996"
Oct 30 20:44:34.195837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:34.195807    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:44:34.203166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:34.203141    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:44:34.976867 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:34.976830    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:35.148798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:35.148754    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:37.116564 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:44:37.116512    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:44:39.935556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:39.935467    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:43.438199 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:43.438171    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:44:43.438611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:43.438248    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:44:44.707506 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:44.707468    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:45.037176 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:45.037145    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:44:49.536793 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:49.536761    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:52.913270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:52.913235    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:54.291772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:54.291741    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:58.889618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:58.889557    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:59.058514 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:44:59.058486    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:00.205783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.205752    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"]
Oct 30 20:45:00.232826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.232797    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.329518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.329478    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"]
Oct 30 20:45:00.343667 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.343639    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:45:00.358603 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.358570    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a103995-df78-4830-801b-3f5f5eddda80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.358752 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.358624    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fljv\" (UniqueName: \"kubernetes.io/projected/7a103995-df78-4830-801b-3f5f5eddda80-kube-api-access-5fljv\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.460328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.460303    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fljv\" (UniqueName: \"kubernetes.io/projected/7a103995-df78-4830-801b-3f5f5eddda80-kube-api-access-5fljv\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.460525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.460513    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a103995-df78-4830-801b-3f5f5eddda80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.461099 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.461082    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a103995-df78-4830-801b-3f5f5eddda80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.517509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.517476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fljv\" (UniqueName: \"kubernetes.io/projected/7a103995-df78-4830-801b-3f5f5eddda80-kube-api-access-5fljv\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.568474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.568447    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:00.950658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:00.950619    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"]
Oct 30 20:45:00.950658 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:45:00.950642    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a103995_df78_4830_801b_3f5f5eddda80.slice/crio-a5c5cb78ced8ce07f5d95fd6f2f092f57fa9a8108b033d9737de464cbe2e6db7 WatchSource:0}: Error finding container a5c5cb78ced8ce07f5d95fd6f2f092f57fa9a8108b033d9737de464cbe2e6db7: Status 404 returned error can't find the container with id a5c5cb78ced8ce07f5d95fd6f2f092f57fa9a8108b033d9737de464cbe2e6db7
Oct 30 20:45:01.806488 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:01.806458    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" event={"ID":"7a103995-df78-4830-801b-3f5f5eddda80","Type":"ContainerStarted","Data":"a5c5cb78ced8ce07f5d95fd6f2f092f57fa9a8108b033d9737de464cbe2e6db7"}
Oct 30 20:45:03.629107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:03.628710    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:05.837478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:05.837447    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" event={"ID":"7a103995-df78-4830-801b-3f5f5eddda80","Type":"ContainerStarted","Data":"b6ec5ae448dd92a9573988a83c5d51b99509d8f5fc667f116abba504be2b00db"}
Oct 30 20:45:05.955617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:05.955565    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" podStartSLOduration=1.512178924 podStartE2EDuration="5.9555517s" totalImagesPullingTime="4.443372776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:05.952841166 +0000 UTC m=+1532.101892249" watchObservedRunningTime="2026-10-30 20:45:05.9555517 +0000 UTC m=+1532.104602687"
Oct 30 20:45:06.846061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:06.846029    5002 generic.go:356] "Generic (PLEG): container finished" podID="7a103995-df78-4830-801b-3f5f5eddda80" containerID="b6ec5ae448dd92a9573988a83c5d51b99509d8f5fc667f116abba504be2b00db" exitCode=60
Oct 30 20:45:06.846466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:06.846126    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" event={"ID":"7a103995-df78-4830-801b-3f5f5eddda80","Type":"ContainerDied","Data":"b6ec5ae448dd92a9573988a83c5d51b99509d8f5fc667f116abba504be2b00db"}
Oct 30 20:45:06.846466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:06.846323    5002 scope.go:122] "RemoveContainer" containerID="b6ec5ae448dd92a9573988a83c5d51b99509d8f5fc667f116abba504be2b00db"
Oct 30 20:45:07.116096 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:45:07.116067    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:45:07.464072 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:07.464043    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:07.855827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:07.855749    5002 generic.go:356] "Generic (PLEG): container finished" podID="7a103995-df78-4830-801b-3f5f5eddda80" containerID="b28886a2371cd2694646204023787a4b923c1c9d6a1c464b69eab60decff0364" exitCode=60
Oct 30 20:45:07.856242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:07.855830    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" event={"ID":"7a103995-df78-4830-801b-3f5f5eddda80","Type":"ContainerDied","Data":"b28886a2371cd2694646204023787a4b923c1c9d6a1c464b69eab60decff0364"}
Oct 30 20:45:07.856242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:07.855885    5002 scope.go:122] "RemoveContainer" containerID="b6ec5ae448dd92a9573988a83c5d51b99509d8f5fc667f116abba504be2b00db"
Oct 30 20:45:07.856242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:07.856124    5002 scope.go:122] "RemoveContainer" containerID="b28886a2371cd2694646204023787a4b923c1c9d6a1c464b69eab60decff0364"
Oct 30 20:45:07.856476 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:45:07.856446    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67_odh-ai-gateway-infra(7a103995-df78-4830-801b-3f5f5eddda80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" podUID="7a103995-df78-4830-801b-3f5f5eddda80"
Oct 30 20:45:08.351971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:08.351930    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:08.561461 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:08.561426    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:08.737918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:08.737886    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:08.866208 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:08.866186    5002 scope.go:122] "RemoveContainer" containerID="b28886a2371cd2694646204023787a4b923c1c9d6a1c464b69eab60decff0364"
Oct 30 20:45:08.866625 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:45:08.866509    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67_odh-ai-gateway-infra(7a103995-df78-4830-801b-3f5f5eddda80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" podUID="7a103995-df78-4830-801b-3f5f5eddda80"
Oct 30 20:45:12.906153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:12.906121    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"]
Oct 30 20:45:12.912236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:12.912209    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.019448 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.019418    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c2bdb58a-serving-cert\""
Oct 30 20:45:13.024187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.024155    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"]
Oct 30 20:45:13.034745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.034711    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.034903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.034864    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbaf9e72-ed04-4da1-b247-3f6732e243e9-openshift-service-ca\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.034903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.034890    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-metrics-certs\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.035234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.034944    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjb9\" (UniqueName: \"kubernetes.io/projected/dbaf9e72-ed04-4da1-b247-3f6732e243e9-kube-api-access-rmjb9\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.136756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.136724    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbaf9e72-ed04-4da1-b247-3f6732e243e9-openshift-service-ca\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.137001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.136975    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-metrics-certs\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.137111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.137055    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjb9\" (UniqueName: \"kubernetes.io/projected/dbaf9e72-ed04-4da1-b247-3f6732e243e9-kube-api-access-rmjb9\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.137257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.137243    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.137454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.137433    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbaf9e72-ed04-4da1-b247-3f6732e243e9-openshift-service-ca\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.137510 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:45:13.137446    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-serving-cert: secret "maas-api-e2e-derive-c2bdb58a-serving-cert" not found
Oct 30 20:45:13.137572 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:45:13.137563    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls podName:dbaf9e72-ed04-4da1-b247-3f6732e243e9 nodeName:}" failed. No retries permitted until 2026-10-30 20:45:13.637522954 +0000 UTC m=+1539.786573932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls") pod "maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf" (UID: "dbaf9e72-ed04-4da1-b247-3f6732e243e9") : secret "maas-api-e2e-derive-c2bdb58a-serving-cert" not found
Oct 30 20:45:13.140443 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.140419    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-metrics-certs\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.177622 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.164276    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:13.198595 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.198512    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjb9\" (UniqueName: \"kubernetes.io/projected/dbaf9e72-ed04-4da1-b247-3f6732e243e9-kube-api-access-rmjb9\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.364459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.364427    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:13.549864 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.549833    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:13.648901 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.648867    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.651504 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.651480    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls\") pod \"maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:13.829260 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:13.829178    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:14.215428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:14.215397    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"]
Oct 30 20:45:14.219797 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:45:14.219697    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaf9e72_ed04_4da1_b247_3f6732e243e9.slice/crio-ae43d399473d72bb531761423479be6cd620e6cc65e235d906d99756d5f8eb39 WatchSource:0}: Error finding container ae43d399473d72bb531761423479be6cd620e6cc65e235d906d99756d5f8eb39: Status 404 returned error can't find the container with id ae43d399473d72bb531761423479be6cd620e6cc65e235d906d99756d5f8eb39
Oct 30 20:45:14.932265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:14.932227    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf" event={"ID":"dbaf9e72-ed04-4da1-b247-3f6732e243e9","Type":"ContainerStarted","Data":"ae43d399473d72bb531761423479be6cd620e6cc65e235d906d99756d5f8eb39"}
Oct 30 20:45:15.137703 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:15.137680    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:45:15.941025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:15.940992    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf" event={"ID":"dbaf9e72-ed04-4da1-b247-3f6732e243e9","Type":"ContainerStarted","Data":"94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa"}
Oct 30 20:45:15.941438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:15.941134    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:16.084553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:16.084293    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf" podStartSLOduration=3.228437453 podStartE2EDuration="4.084272764s" totalImagesPullingTime="855.835311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:16.072808016 +0000 UTC m=+1542.221859002" watchObservedRunningTime="2026-10-30 20:45:16.084272764 +0000 UTC m=+1542.233323751"
Oct 30 20:45:18.209587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:18.207260    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:18.497669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:18.497565    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:45:19.967435 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:19.967408    5002 scope.go:122] "RemoveContainer" containerID="b28886a2371cd2694646204023787a4b923c1c9d6a1c464b69eab60decff0364"
Oct 30 20:45:21.122378 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:21.122347    5002 generic.go:356] "Generic (PLEG): container finished" podID="7a103995-df78-4830-801b-3f5f5eddda80" containerID="8ef3641222c53eb5d89583474138ef40ea375bd6f73f30b41784d717d1a0fea9" exitCode=60
Oct 30 20:45:21.122835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:21.122420    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" event={"ID":"7a103995-df78-4830-801b-3f5f5eddda80","Type":"ContainerDied","Data":"8ef3641222c53eb5d89583474138ef40ea375bd6f73f30b41784d717d1a0fea9"}
Oct 30 20:45:21.122835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:21.122504    5002 scope.go:122] "RemoveContainer" containerID="b28886a2371cd2694646204023787a4b923c1c9d6a1c464b69eab60decff0364"
Oct 30 20:45:21.122955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:21.122890    5002 scope.go:122] "RemoveContainer" containerID="8ef3641222c53eb5d89583474138ef40ea375bd6f73f30b41784d717d1a0fea9"
Oct 30 20:45:21.123277 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:45:21.123243    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67_odh-ai-gateway-infra(7a103995-df78-4830-801b-3f5f5eddda80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" podUID="7a103995-df78-4830-801b-3f5f5eddda80"
Oct 30 20:45:21.957627 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:21.957600    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:45:22.255896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.255867    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"]
Oct 30 20:45:22.424926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.424901    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:22.567342 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.567261    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a103995-df78-4830-801b-3f5f5eddda80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a103995-df78-4830-801b-3f5f5eddda80-maas-api-service-ca\") pod \"7a103995-df78-4830-801b-3f5f5eddda80\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") "
Oct 30 20:45:22.567517 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.567361    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a103995-df78-4830-801b-3f5f5eddda80-kube-api-access-5fljv\" (UniqueName: \"kubernetes.io/projected/7a103995-df78-4830-801b-3f5f5eddda80-kube-api-access-5fljv\") pod \"7a103995-df78-4830-801b-3f5f5eddda80\" (UID: \"7a103995-df78-4830-801b-3f5f5eddda80\") "
Oct 30 20:45:22.567686 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.567658    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a103995-df78-4830-801b-3f5f5eddda80-maas-api-service-ca" pod "7a103995-df78-4830-801b-3f5f5eddda80" (UID: "7a103995-df78-4830-801b-3f5f5eddda80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:22.568285 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.568263    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a103995-df78-4830-801b-3f5f5eddda80-maas-api-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:22.569852 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.569823    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a103995-df78-4830-801b-3f5f5eddda80-kube-api-access-5fljv" pod "7a103995-df78-4830-801b-3f5f5eddda80" (UID: "7a103995-df78-4830-801b-3f5f5eddda80"). InnerVolumeSpecName "kube-api-access-5fljv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:22.670122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:22.670088    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fljv\" (UniqueName: \"kubernetes.io/projected/7a103995-df78-4830-801b-3f5f5eddda80-kube-api-access-5fljv\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:23.017856 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:23.017808    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:23.145178 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:23.145140    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67" event={"ID":"7a103995-df78-4830-801b-3f5f5eddda80","Type":"ContainerDied","Data":"a5c5cb78ced8ce07f5d95fd6f2f092f57fa9a8108b033d9737de464cbe2e6db7"}
Oct 30 20:45:23.145178 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:23.145178    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"
Oct 30 20:45:23.145414 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:23.145207    5002 scope.go:122] "RemoveContainer" containerID="8ef3641222c53eb5d89583474138ef40ea375bd6f73f30b41784d717d1a0fea9"
Oct 30 20:45:23.314518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:23.314432    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"]
Oct 30 20:45:23.319092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:23.319066    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889885-mbh67"]
Oct 30 20:45:23.969289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:23.969258    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a103995-df78-4830-801b-3f5f5eddda80" path="/var/lib/kubelet/pods/7a103995-df78-4830-801b-3f5f5eddda80/volumes"
Oct 30 20:45:27.727288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:27.727251    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:32.479301 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:32.479261    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:32.863307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:32.863232    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:33.049420 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:33.049356    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:33.670616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:33.668665    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:34.123576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:34.123511    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:35.305704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:35.305671    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:36.060568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:36.043523    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:36.595070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:36.595032    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:37.116296 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:45:37.116259    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:45:37.478589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:37.478560    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:37.721557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:37.714229    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:40.736115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:40.736062    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:40.974725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:40.969353    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:41.289108 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.289068    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"]
Oct 30 20:45:41.290406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.290375    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:41.290602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.290584    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:41.290822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.290807    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:41.290920 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.290909    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:41.291163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.291145    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:41.291359 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.291345    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:41.291466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.291454    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:41.314109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.314086    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.407422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.407378    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372d973-3a84-4120-a0f2-a70f3355179c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") " pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.457887 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.457849    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f278m\" (UniqueName: \"kubernetes.io/projected/7372d973-3a84-4120-a0f2-a70f3355179c-kube-api-access-f278m\") pod \"payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") " pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.459664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.458370    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c650f2b7\""
Oct 30 20:45:41.460027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.458425    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c650f2b7-dockercfg-d6jx2\""
Oct 30 20:45:41.460155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.432050    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:41.551757 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.551670    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"]
Oct 30 20:45:41.563551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.560730    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372d973-3a84-4120-a0f2-a70f3355179c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") " pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.563551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.560796    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f278m\" (UniqueName: \"kubernetes.io/projected/7372d973-3a84-4120-a0f2-a70f3355179c-kube-api-access-f278m\") pod \"payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") " pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.563551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.562162    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372d973-3a84-4120-a0f2-a70f3355179c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") " pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.646266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.646242    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f278m\" (UniqueName: \"kubernetes.io/projected/7372d973-3a84-4120-a0f2-a70f3355179c-kube-api-access-f278m\") pod \"payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") " pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.698674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.698645    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:41.751767 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.751742    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:41.900105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:41.900026    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:42.175065 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:42.175041    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"]
Oct 30 20:45:42.178063 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:45:42.177790    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7372d973_3a84_4120_a0f2_a70f3355179c.slice/crio-27a9223fd120b191779a264d09aa747aaa84fde8e76b43fa873aafe812edbcca WatchSource:0}: Error finding container 27a9223fd120b191779a264d09aa747aaa84fde8e76b43fa873aafe812edbcca: Status 404 returned error can't find the container with id 27a9223fd120b191779a264d09aa747aaa84fde8e76b43fa873aafe812edbcca
Oct 30 20:45:42.591881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:42.591849    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg" event={"ID":"7372d973-3a84-4120-a0f2-a70f3355179c","Type":"ContainerStarted","Data":"56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca"}
Oct 30 20:45:42.592056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:42.591921    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg" event={"ID":"7372d973-3a84-4120-a0f2-a70f3355179c","Type":"ContainerStarted","Data":"27a9223fd120b191779a264d09aa747aaa84fde8e76b43fa873aafe812edbcca"}
Oct 30 20:45:42.592056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:42.591993    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:42.737262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:42.737209    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg" podStartSLOduration=1.737195515 podStartE2EDuration="1.737195515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:42.732856168 +0000 UTC m=+1568.881907157" watchObservedRunningTime="2026-10-30 20:45:42.737195515 +0000 UTC m=+1568.886246502"
Oct 30 20:45:42.958625 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:42.958436    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:45.900367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:45.900335    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:46.110554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.110500    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:46.219565 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.219522    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"]
Oct 30 20:45:46.220617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.220592    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:46.220769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.220755    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a103995-df78-4830-801b-3f5f5eddda80" containerName="cleanup"
Oct 30 20:45:46.225510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.225487    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:46.283957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.283923    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"]
Oct 30 20:45:46.329971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.329866    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-d71c23-dockercfg-qgpz5\""
Oct 30 20:45:46.329971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.329940    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-d71c23\""
Oct 30 20:45:46.404556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.387155    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9smq\" (UniqueName: \"kubernetes.io/projected/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-kube-api-access-t9smq\") pod \"payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:46.404556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.387225    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:46.502662 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.495509    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9smq\" (UniqueName: \"kubernetes.io/projected/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-kube-api-access-t9smq\") pod \"payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:46.502662 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.495601    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:46.511950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.511862    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:46.572477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.572444    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9smq\" (UniqueName: \"kubernetes.io/projected/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-kube-api-access-t9smq\") pod \"payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:46.842287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:46.842198    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:47.241987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:47.241958    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"]
Oct 30 20:45:47.635252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:47.635169    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq" event={"ID":"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77","Type":"ContainerStarted","Data":"99615a8d876011af3710c7557fe34fb4154cab13ca1737a28df4aaae64bda276"}
Oct 30 20:45:47.635519 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:47.635496    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq" event={"ID":"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77","Type":"ContainerStarted","Data":"af7f89e372b75dcde68540643f3c288c21556481ee405fc599b54ef4be327eb7"}
Oct 30 20:45:47.635733 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:47.635708    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:45:47.755734 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:47.755664    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq" podStartSLOduration=1.75564524 podStartE2EDuration="1.75564524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:47.752264303 +0000 UTC m=+1573.901315291" watchObservedRunningTime="2026-10-30 20:45:47.75564524 +0000 UTC m=+1573.904696227"
Oct 30 20:45:48.111142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:48.111061    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:49.652243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:49.652213    5002 generic.go:356] "Generic (PLEG): container finished" podID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerID="9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564" exitCode=0
Oct 30 20:45:49.652811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:49.652295    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" event={"ID":"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d","Type":"ContainerDied","Data":"9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564"}
Oct 30 20:45:49.652811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:49.652774    5002 scope.go:122] "RemoveContainer" containerID="9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564"
Oct 30 20:45:49.654336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:49.654308    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:49.654432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:49.654382    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:49.654432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:49.654413    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:50.661971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:50.661935    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" event={"ID":"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d","Type":"ContainerStarted","Data":"c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79"}
Oct 30 20:45:50.985517 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:50.985489    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:45:50.985714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:50.985651    5002 patch_prober.go:29] interesting pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:45:50.985714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:50.985695    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:45:51.985245 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:51.985211    5002 patch_prober.go:29] interesting pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:45:51.985766 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:51.985269    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:45:52.819329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:52.819159    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:52.985390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:52.985325    5002 patch_prober.go:29] interesting pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:45:52.985921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:52.985395    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:45:53.603615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:53.603581    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:45:53.985699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:53.985662    5002 patch_prober.go:29] interesting pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:45:53.986109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:53.985720    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:45:54.985955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:54.985923    5002 patch_prober.go:29] interesting pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:45:54.986414 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:54.985971    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:45:55.989894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:55.989867    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:45:55.990368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:55.990350    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:45:55.991013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:55.990994    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:45:57.496721 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:57.496634    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:58.646264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:45:58.646231    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:46:02.357491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:02.357455    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:07.147154 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:46:07.147112    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:46:07.635835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:07.635799    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:09.594717 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:09.594683    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:46:10.973132 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:10.973101    5002 patch_prober.go:29] interesting pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": read tcp 10.132.0.2:34334->10.132.0.99:15021: read: connection reset by peer" start-of-body=
Oct 30 20:46:10.973623 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:10.973154    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": read tcp 10.132.0.2:34334->10.132.0.99:15021: read: connection reset by peer"
Oct 30 20:46:11.949066 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:11.949035    5002 generic.go:356] "Generic (PLEG): container finished" podID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerID="85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849" exitCode=0
Oct 30 20:46:11.949244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:11.949107    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" event={"ID":"e8db72a9-39e5-4274-aff2-856577cfe58d","Type":"ContainerDied","Data":"85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849"}
Oct 30 20:46:11.949500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:11.949479    5002 scope.go:122] "RemoveContainer" containerID="85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849"
Oct 30 20:46:11.950864 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:11.950834    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:11.950953 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:11.950903    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:11.950953 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:11.950944    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:12.812559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:12.803903    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:12.958866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:12.958832    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" event={"ID":"e8db72a9-39e5-4274-aff2-856577cfe58d","Type":"ContainerStarted","Data":"661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f"}
Oct 30 20:46:13.350689 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:13.350649    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:46:13.350895 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:13.350869    5002 patch_prober.go:29] interesting pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:13.350951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:13.350929    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:14.350632 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:14.350601    5002 patch_prober.go:29] interesting pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:14.351052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:14.350688    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:14.581779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:14.581744    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:15.333628 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:15.332139    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:15.356825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:15.356793    5002 patch_prober.go:29] interesting pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:15.357459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:15.357410    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:15.545520 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:15.545481    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:16.146190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:16.146149    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"]
Oct 30 20:46:16.263846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:16.263817    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"]
Oct 30 20:46:16.350777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:16.350748    5002 patch_prober.go:29] interesting pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:16.350956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:16.350803    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:16.547202 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:16.547169    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:16.735159 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:16.734024    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:17.351144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:17.351111    5002 patch_prober.go:29] interesting pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:17.351338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:17.351162    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:18.265644 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:18.265603    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:18.432458 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:18.432427    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:46:18.432990 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:18.432956    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:46:18.433850 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:18.433834    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:46:19.855141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:19.855097    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:20.076697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:20.076623    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:20.417713 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:20.417670    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:21.147456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.147422    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx" podUID="676ff44f-01bf-428f-a784-d829bec9499b" containerName="payload-processing" containerID="cri-o://550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b" gracePeriod=25
Oct 30 20:46:21.264403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.264352    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7" podUID="e094724d-5b57-4ff1-a451-05be1852311c" containerName="payload-pre-processing" containerID="cri-o://e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c" gracePeriod=25
Oct 30 20:46:21.475179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.475150    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:46:21.560229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.560204    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676ff44f-01bf-428f-a784-d829bec9499b-kube-api-access-xq9wt\" (UniqueName: \"kubernetes.io/projected/676ff44f-01bf-428f-a784-d829bec9499b-kube-api-access-xq9wt\") pod \"676ff44f-01bf-428f-a784-d829bec9499b\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") "
Oct 30 20:46:21.560375 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.560261    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676ff44f-01bf-428f-a784-d829bec9499b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ff44f-01bf-428f-a784-d829bec9499b-plugins-config-volume\") pod \"676ff44f-01bf-428f-a784-d829bec9499b\" (UID: \"676ff44f-01bf-428f-a784-d829bec9499b\") "
Oct 30 20:46:21.560643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.560618    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676ff44f-01bf-428f-a784-d829bec9499b-plugins-config-volume" pod "676ff44f-01bf-428f-a784-d829bec9499b" (UID: "676ff44f-01bf-428f-a784-d829bec9499b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:21.561097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.561080    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676ff44f-01bf-428f-a784-d829bec9499b-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:21.562731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.562705    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676ff44f-01bf-428f-a784-d829bec9499b-kube-api-access-xq9wt" pod "676ff44f-01bf-428f-a784-d829bec9499b" (UID: "676ff44f-01bf-428f-a784-d829bec9499b"). InnerVolumeSpecName "kube-api-access-xq9wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:21.572676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.572655    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:46:21.662281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.662197    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e094724d-5b57-4ff1-a451-05be1852311c-kube-api-access-s2z9m\" (UniqueName: \"kubernetes.io/projected/e094724d-5b57-4ff1-a451-05be1852311c-kube-api-access-s2z9m\") pod \"e094724d-5b57-4ff1-a451-05be1852311c\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") "
Oct 30 20:46:21.662281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.662239    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e094724d-5b57-4ff1-a451-05be1852311c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e094724d-5b57-4ff1-a451-05be1852311c-plugins-config-volume\") pod \"e094724d-5b57-4ff1-a451-05be1852311c\" (UID: \"e094724d-5b57-4ff1-a451-05be1852311c\") "
Oct 30 20:46:21.662633 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.662611    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e094724d-5b57-4ff1-a451-05be1852311c-plugins-config-volume" pod "e094724d-5b57-4ff1-a451-05be1852311c" (UID: "e094724d-5b57-4ff1-a451-05be1852311c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:21.662975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.662955    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e094724d-5b57-4ff1-a451-05be1852311c-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:21.663050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.662979    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9wt\" (UniqueName: \"kubernetes.io/projected/676ff44f-01bf-428f-a784-d829bec9499b-kube-api-access-xq9wt\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:21.664404 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.664383    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e094724d-5b57-4ff1-a451-05be1852311c-kube-api-access-s2z9m" pod "e094724d-5b57-4ff1-a451-05be1852311c" (UID: "e094724d-5b57-4ff1-a451-05be1852311c"). InnerVolumeSpecName "kube-api-access-s2z9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:21.764941 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:21.764910    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2z9m\" (UniqueName: \"kubernetes.io/projected/e094724d-5b57-4ff1-a451-05be1852311c-kube-api-access-s2z9m\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:22.069475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.069440    5002 generic.go:356] "Generic (PLEG): container finished" podID="e094724d-5b57-4ff1-a451-05be1852311c" containerID="e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c" exitCode=0
Oct 30 20:46:22.069659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.069490    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"
Oct 30 20:46:22.069659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.069521    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7" event={"ID":"e094724d-5b57-4ff1-a451-05be1852311c","Type":"ContainerDied","Data":"e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c"}
Oct 30 20:46:22.069659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.069611    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7" event={"ID":"e094724d-5b57-4ff1-a451-05be1852311c","Type":"ContainerDied","Data":"74c31dab32ecda5ce4cc40f84f985d404f2ff253bc863ea30a8bf58484fe7f4f"}
Oct 30 20:46:22.069659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.069638    5002 scope.go:122] "RemoveContainer" containerID="e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c"
Oct 30 20:46:22.071312 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.071287    5002 generic.go:356] "Generic (PLEG): container finished" podID="676ff44f-01bf-428f-a784-d829bec9499b" containerID="550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b" exitCode=0
Oct 30 20:46:22.071454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.071347    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"
Oct 30 20:46:22.071454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.071436    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx" event={"ID":"676ff44f-01bf-428f-a784-d829bec9499b","Type":"ContainerDied","Data":"550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b"}
Oct 30 20:46:22.071695 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.071520    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx" event={"ID":"676ff44f-01bf-428f-a784-d829bec9499b","Type":"ContainerDied","Data":"1e145959b654042800210fba58dd5a113c06d4016f81d2fc47e24463354abc9b"}
Oct 30 20:46:22.080415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.080400    5002 scope.go:122] "RemoveContainer" containerID="e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c"
Oct 30 20:46:22.080754 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:46:22.080733    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c\": container with ID starting with e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c not found: ID does not exist" containerID="e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c"
Oct 30 20:46:22.080832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.080760    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c"} err="failed to get container status \"e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c\": rpc error: code = NotFound desc = could not find container \"e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c\": container with ID starting with e0ff52873e75d596e3da8edbebc1a535f4d80b2e57305de7943d799df649065c not found: ID does not exist"
Oct 30 20:46:22.080832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.080774    5002 scope.go:122] "RemoveContainer" containerID="550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b"
Oct 30 20:46:22.091014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.090978    5002 scope.go:122] "RemoveContainer" containerID="550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b"
Oct 30 20:46:22.091246 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:46:22.091226    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b\": container with ID starting with 550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b not found: ID does not exist" containerID="550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b"
Oct 30 20:46:22.091312 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.091256    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b"} err="failed to get container status \"550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b\": rpc error: code = NotFound desc = could not find container \"550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b\": container with ID starting with 550e5c70c4486856230611b5e8354924e3d0c1c99ed60cd7a0713420a43bab5b not found: ID does not exist"
Oct 30 20:46:22.237259 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.237198    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"]
Oct 30 20:46:22.288007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.287977    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7"]
Oct 30 20:46:22.354518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.354492    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"]
Oct 30 20:46:22.361213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:22.361190    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx"]
Oct 30 20:46:23.143326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:23.143295    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:23.341034 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:23.341005    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:23.539902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:23.539872    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:23.867678 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:23.867601    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:23.973520 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:23.973487    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676ff44f-01bf-428f-a784-d829bec9499b" path="/var/lib/kubelet/pods/676ff44f-01bf-428f-a784-d829bec9499b/volumes"
Oct 30 20:46:23.974046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:23.974024    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e094724d-5b57-4ff1-a451-05be1852311c" path="/var/lib/kubelet/pods/e094724d-5b57-4ff1-a451-05be1852311c/volumes"
Oct 30 20:46:24.072843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:24.072807    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:24.262336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:24.262307    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:26.841363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:26.841327    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:27.462763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.462725    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:27.646839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.646808    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:27.831233 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.831087    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"]
Oct 30 20:46:27.832889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.832848    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e094724d-5b57-4ff1-a451-05be1852311c" containerName="payload-pre-processing"
Oct 30 20:46:27.833073 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.833054    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e094724d-5b57-4ff1-a451-05be1852311c" containerName="payload-pre-processing"
Oct 30 20:46:27.833245 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.833230    5002 cpu_manager.go:414] "containerMap: removing container" podUID="676ff44f-01bf-428f-a784-d829bec9499b" containerName="payload-processing"
Oct 30 20:46:27.833388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.833371    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676ff44f-01bf-428f-a784-d829bec9499b" containerName="payload-processing"
Oct 30 20:46:27.833776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.833753    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="676ff44f-01bf-428f-a784-d829bec9499b" containerName="payload-processing"
Oct 30 20:46:27.833914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.833902    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e094724d-5b57-4ff1-a451-05be1852311c" containerName="payload-pre-processing"
Oct 30 20:46:27.840835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.840814    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:27.893344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.893315    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"]
Oct 30 20:46:27.945559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.941838    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-27219c73\""
Oct 30 20:46:27.945559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.942011    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-27219c73-dockercfg-g2fcc\""
Oct 30 20:46:27.985561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:27.977611    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv66k\" (UniqueName: \"kubernetes.io/projected/fb912d4f-67f8-4d21-83d5-64193673e2c2-kube-api-access-xv66k\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:28.112250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.112161    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb912d4f-67f8-4d21-83d5-64193673e2c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:28.126042 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.126013    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"]
Oct 30 20:46:28.134813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.134782    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.155025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.154841    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"]
Oct 30 20:46:28.215154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.215124    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv66k\" (UniqueName: \"kubernetes.io/projected/fb912d4f-67f8-4d21-83d5-64193673e2c2-kube-api-access-xv66k\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:28.215397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.215382    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb912d4f-67f8-4d21-83d5-64193673e2c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:28.215470 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.215452    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c4bbd4-c505-4b24-8f91-205f82874461-plugins-config-volume\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-mxwzh\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.215526 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.215515    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxj9\" (UniqueName: \"kubernetes.io/projected/18c4bbd4-c505-4b24-8f91-205f82874461-kube-api-access-7zxj9\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-mxwzh\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.216310 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.216285    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb912d4f-67f8-4d21-83d5-64193673e2c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:28.273661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.273637    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv66k\" (UniqueName: \"kubernetes.io/projected/fb912d4f-67f8-4d21-83d5-64193673e2c2-kube-api-access-xv66k\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:28.317465 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.317438    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c4bbd4-c505-4b24-8f91-205f82874461-plugins-config-volume\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-mxwzh\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.317617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.317481    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxj9\" (UniqueName: \"kubernetes.io/projected/18c4bbd4-c505-4b24-8f91-205f82874461-kube-api-access-7zxj9\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-mxwzh\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.318287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.318262    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c4bbd4-c505-4b24-8f91-205f82874461-plugins-config-volume\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-mxwzh\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.376198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.376143    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxj9\" (UniqueName: \"kubernetes.io/projected/18c4bbd4-c505-4b24-8f91-205f82874461-kube-api-access-7zxj9\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-mxwzh\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.455875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.455843    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:28.464227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.464204    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:28.668430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.666337    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:28.868746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.868718    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"]
Oct 30 20:46:28.870999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.870976    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:28.921130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:28.921070    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"]
Oct 30 20:46:28.932882 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:46:28.932853    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c4bbd4_c505_4b24_8f91_205f82874461.slice/crio-fa4809c4729d12ebbd7912ee787d4c306c7bea569b5f6b84b4ed148270d1027b WatchSource:0}: Error finding container fa4809c4729d12ebbd7912ee787d4c306c7bea569b5f6b84b4ed148270d1027b: Status 404 returned error can't find the container with id fa4809c4729d12ebbd7912ee787d4c306c7bea569b5f6b84b4ed148270d1027b
Oct 30 20:46:29.060733 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.060671    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:29.178445 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.178359    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" event={"ID":"18c4bbd4-c505-4b24-8f91-205f82874461","Type":"ContainerStarted","Data":"c13c0688ed3983be2b2d93d3415d2c4107ec37a4c58eac8bfbbe4d57999bbb91"}
Oct 30 20:46:29.178612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.178450    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" event={"ID":"18c4bbd4-c505-4b24-8f91-205f82874461","Type":"ContainerStarted","Data":"fa4809c4729d12ebbd7912ee787d4c306c7bea569b5f6b84b4ed148270d1027b"}
Oct 30 20:46:29.178612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.178480    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:29.180629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.180562    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" event={"ID":"fb912d4f-67f8-4d21-83d5-64193673e2c2","Type":"ContainerStarted","Data":"e8ac986545159b4295b31a0e6aaf566c9269d51243ec587e0389bdf0d88e6b9a"}
Oct 30 20:46:29.180764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.180660    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" event={"ID":"fb912d4f-67f8-4d21-83d5-64193673e2c2","Type":"ContainerStarted","Data":"d3fb06a6a7f81151738836c9bd60c40d2732e174778c05d1ad9548baa2691f61"}
Oct 30 20:46:29.180764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.180717    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:29.446796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.444188    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" podStartSLOduration=2.444163986 podStartE2EDuration="2.444163986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:29.422908363 +0000 UTC m=+1615.571959351" watchObservedRunningTime="2026-10-30 20:46:29.444163986 +0000 UTC m=+1615.593215003"
Oct 30 20:46:29.446796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:29.445462    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" podStartSLOduration=2.445451277 podStartE2EDuration="2.445451277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:29.303460503 +0000 UTC m=+1615.452511491" watchObservedRunningTime="2026-10-30 20:46:29.445451277 +0000 UTC m=+1615.594502265"
Oct 30 20:46:33.196683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:33.196633    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:33.372679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:33.372647    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:33.614446 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:33.614408    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:37.116692 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:46:37.116653    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:46:38.285330 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:38.285256    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:38.477000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:38.476964    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:38.785148 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:38.785118    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:40.191888 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:40.191863    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:46:40.192289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:40.191965    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:46:43.314636 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:43.314597    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:48.090039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:48.090004    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:50.733406 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:50.733364    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:52.791888 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:52.791847    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:56.780788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:56.780752    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:57.055051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:57.054957    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:57.982686 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:57.982650    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"]
Oct 30 20:46:57.983109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:57.982875    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf" podUID="dbaf9e72-ed04-4da1-b247-3f6732e243e9" containerName="maas-api" containerID="cri-o://94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa" gracePeriod=30
Oct 30 20:46:58.132028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.132000    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:58.311262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.311232    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:46:58.444597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.444517    5002 generic.go:356] "Generic (PLEG): container finished" podID="dbaf9e72-ed04-4da1-b247-3f6732e243e9" containerID="94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa" exitCode=0
Oct 30 20:46:58.444853 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.444836    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf" event={"ID":"dbaf9e72-ed04-4da1-b247-3f6732e243e9","Type":"ContainerDied","Data":"94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa"}
Oct 30 20:46:58.444930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.444897    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf" event={"ID":"dbaf9e72-ed04-4da1-b247-3f6732e243e9","Type":"ContainerDied","Data":"ae43d399473d72bb531761423479be6cd620e6cc65e235d906d99756d5f8eb39"}
Oct 30 20:46:58.444930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.444927    5002 scope.go:122] "RemoveContainer" containerID="94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa"
Oct 30 20:46:58.445094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.445079    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"
Oct 30 20:46:58.486724 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.486700    5002 scope.go:122] "RemoveContainer" containerID="94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa"
Oct 30 20:46:58.487496 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:46:58.487406    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa\": container with ID starting with 94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa not found: ID does not exist" containerID="94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa"
Oct 30 20:46:58.487734 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.487700    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa"} err="failed to get container status \"94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa\": rpc error: code = NotFound desc = could not find container \"94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa\": container with ID starting with 94178132a3766aaccab3e741d65a0d88703f2e2c5a80028aba2696f73dbb7faa not found: ID does not exist"
Oct 30 20:46:58.505366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.505332    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbaf9e72-ed04-4da1-b247-3f6732e243e9-kube-api-access-rmjb9\" (UniqueName: \"kubernetes.io/projected/dbaf9e72-ed04-4da1-b247-3f6732e243e9-kube-api-access-rmjb9\") pod \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") "
Oct 30 20:46:58.505522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.505446    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls\") pod \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") "
Oct 30 20:46:58.505611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.505583    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-metrics-certs\") pod \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") "
Oct 30 20:46:58.505685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.505652    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbaf9e72-ed04-4da1-b247-3f6732e243e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbaf9e72-ed04-4da1-b247-3f6732e243e9-openshift-service-ca\") pod \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\" (UID: \"dbaf9e72-ed04-4da1-b247-3f6732e243e9\") "
Oct 30 20:46:58.506613 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.506574    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbaf9e72-ed04-4da1-b247-3f6732e243e9-openshift-service-ca" pod "dbaf9e72-ed04-4da1-b247-3f6732e243e9" (UID: "dbaf9e72-ed04-4da1-b247-3f6732e243e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:58.515902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.515863    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-metrics-certs" pod "dbaf9e72-ed04-4da1-b247-3f6732e243e9" (UID: "dbaf9e72-ed04-4da1-b247-3f6732e243e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:58.521958 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.521920    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls" pod "dbaf9e72-ed04-4da1-b247-3f6732e243e9" (UID: "dbaf9e72-ed04-4da1-b247-3f6732e243e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:58.523077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.523048    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbaf9e72-ed04-4da1-b247-3f6732e243e9-kube-api-access-rmjb9" pod "dbaf9e72-ed04-4da1-b247-3f6732e243e9" (UID: "dbaf9e72-ed04-4da1-b247-3f6732e243e9"). InnerVolumeSpecName "kube-api-access-rmjb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:58.607990 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.607960    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjb9\" (UniqueName: \"kubernetes.io/projected/dbaf9e72-ed04-4da1-b247-3f6732e243e9-kube-api-access-rmjb9\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:58.607990 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.607988    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:58.607990 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.607997    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaf9e72-ed04-4da1-b247-3f6732e243e9-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:58.608220 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.608005    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbaf9e72-ed04-4da1-b247-3f6732e243e9-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:58.763458 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.763423    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:58.869903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.869833    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"]
Oct 30 20:46:58.873583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:58.873564    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bdb58a-7c7f4dcb57-fpmqf"]
Oct 30 20:46:59.087556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:59.086648    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:59.873428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:59.873395    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:59.967072 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:46:59.967036    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbaf9e72-ed04-4da1-b247-3f6732e243e9" path="/var/lib/kubelet/pods/dbaf9e72-ed04-4da1-b247-3f6732e243e9/volumes"
Oct 30 20:47:00.067911 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.067879    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:00.253894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.253848    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:00.553893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.553807    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"]
Oct 30 20:47:00.554892 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.554081    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy" containerID="cri-o://c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79" gracePeriod=30
Oct 30 20:47:00.641367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.641334    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:00.869004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.868893    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:00.872251 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.872224    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"]
Oct 30 20:47:00.872496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:00.872460    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" containerID="cri-o://661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f" gracePeriod=30
Oct 30 20:47:01.392930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.392892    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:01.640115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.640073    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:01.877584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.877523    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"]
Oct 30 20:47:01.878501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.878477    5002 cpu_manager.go:414] "containerMap: removing container" podUID="dbaf9e72-ed04-4da1-b247-3f6732e243e9" containerName="maas-api"
Oct 30 20:47:01.878501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.878501    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbaf9e72-ed04-4da1-b247-3f6732e243e9" containerName="maas-api"
Oct 30 20:47:01.878718 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.878707    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbaf9e72-ed04-4da1-b247-3f6732e243e9" containerName="maas-api"
Oct 30 20:47:01.884031 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.883980    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.893238 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893163    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-certs\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.893642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893388    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-envoy\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.893753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893676    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-socket\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.893753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893753    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-credential-socket\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.893917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893808    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-token\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.893917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893832    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4qj\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-kube-api-access-sv4qj\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.893917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893888    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istiod-ca-cert\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.908950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.893995    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-data\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.908950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.897893    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-podinfo\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:01.964557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.958216    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"]
Oct 30 20:47:02.010561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:01.997751    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-0ac4e5-openshift-default-dockercfg-ksphv\""
Oct 30 20:47:02.039558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.016798    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-certs\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.061456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.056605    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-envoy\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.062043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.062011    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-socket\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.062256 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.062240    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-credential-socket\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.062390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.062373    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-token\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.062495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.062479    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4qj\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-kube-api-access-sv4qj\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.062671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.062653    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istiod-ca-cert\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.062916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.062900    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-data\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.063027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.063014    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-podinfo\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.063652 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.020461    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-certs\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.095554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.090666    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-credential-socket\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.095554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.091728    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istiod-ca-cert\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.099582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.097175    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-socket\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.099582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.098654    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-data\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.099781 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.099722    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-envoy\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.101225 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.100425    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:02.103961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.103940    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-podinfo\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.149891 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.149405    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-token\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.150313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.150275    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4qj\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-kube-api-access-sv4qj\") pod \"e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") " pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.257557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.257473    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:02.476557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.457085    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:02.659127 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.659035    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:02.799867 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:02.799838    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a71733_90b6_46c8_8c95_80e0ed2d9251.slice/crio-43e3775db18abbe892a9b727da62c8eecc1d10533cb9c67ba9ee293601d10e89 WatchSource:0}: Error finding container 43e3775db18abbe892a9b727da62c8eecc1d10533cb9c67ba9ee293601d10e89: Status 404 returned error can't find the container with id 43e3775db18abbe892a9b727da62c8eecc1d10533cb9c67ba9ee293601d10e89
Oct 30 20:47:02.800749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.800683    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"]
Oct 30 20:47:02.803866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.803824    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:02.804010 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.803987    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:02.804081 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.804063    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:02.936663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:02.936588    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:03.435457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:03.435422    5002 patch_prober.go:29] interesting pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 20:47:03.435680 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:03.435486    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:47:03.465947 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:03.465921    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:03.502169 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:03.502144    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" event={"ID":"d5a71733-90b6-46c8-8c95-80e0ed2d9251","Type":"ContainerStarted","Data":"51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f"}
Oct 30 20:47:03.502521 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:03.502187    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" event={"ID":"d5a71733-90b6-46c8-8c95-80e0ed2d9251","Type":"ContainerStarted","Data":"43e3775db18abbe892a9b727da62c8eecc1d10533cb9c67ba9ee293601d10e89"}
Oct 30 20:47:03.624896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:03.624828    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podStartSLOduration=2.624810331 podStartE2EDuration="2.624810331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:03.621154474 +0000 UTC m=+1649.770205473" watchObservedRunningTime="2026-10-30 20:47:03.624810331 +0000 UTC m=+1649.773861319"
Oct 30 20:47:04.259386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:04.259352    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:04.267285 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:04.267256    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:04.267634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:04.267616    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:47:04.511902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:04.511820    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:04.512814 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:04.512792    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:47:05.858671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.858645    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:47:05.934884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.934857    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istiod-ca-cert\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.934998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.934920    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-kube-api-access-dnvsc\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-kube-api-access-dnvsc\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.934998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.934976    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-podinfo\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.935097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935048    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-credential-socket\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.935171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935157    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-data\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.935249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935221    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istiod-ca-cert" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:05.935387 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935252    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-envoy\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.935473 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935429    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-certs\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.935544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935341    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-credential-socket" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:05.935544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935401    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-data" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:05.935647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935549    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-token\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-token\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.935647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935627    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-socket\") pod \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\" (UID: \"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d\") "
Oct 30 20:47:05.935964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.935896    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-certs" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:05.936106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.936081    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-socket" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:05.936375 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.936352    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:05.936491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.936382    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:05.936491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.936397    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:05.936491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.936409    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:05.936491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.936421    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:05.937337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.937320    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-kube-api-access-dnvsc" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "kube-api-access-dnvsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:05.937426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.937414    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-envoy" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:05.937911 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.937892    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-podinfo" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:05.938239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:05.938220    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-token" pod "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" (UID: "c88c8a7e-8ab5-466b-abfb-2ca8de7a914d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:06.042577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.042499    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.042577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.042548    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.042577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.042562    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvsc\" (UniqueName: \"kubernetes.io/projected/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-kube-api-access-dnvsc\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.042577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.042575    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.140703 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.135581    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:06.184499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.184475    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:47:06.245465 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245437    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-kube-api-access-967wz\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-kube-api-access-967wz\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.245675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245548    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-data\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.245675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245601    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-credential-socket\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.245675 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245635    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-envoy\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.245835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245715    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-certs\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.245835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245754    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-data" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:06.245926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245907    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-credential-socket" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:06.245979 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245945    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-socket\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.246035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.245976    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-podinfo\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.246035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246001    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-certs" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:06.246035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246008    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-token\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-token\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.246182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246097    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8db72a9-39e5-4274-aff2-856577cfe58d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8db72a9-39e5-4274-aff2-856577cfe58d-istiod-ca-cert\") pod \"e8db72a9-39e5-4274-aff2-856577cfe58d\" (UID: \"e8db72a9-39e5-4274-aff2-856577cfe58d\") "
Oct 30 20:47:06.246182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246163    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-socket" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:06.246515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246493    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8db72a9-39e5-4274-aff2-856577cfe58d-istiod-ca-cert" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:06.246748 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246726    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.246886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246751    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.246886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246762    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.246886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246770    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.246886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.246778    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8db72a9-39e5-4274-aff2-856577cfe58d-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.247946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.247924    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-kube-api-access-967wz" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "kube-api-access-967wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:06.248019 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.247962    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-envoy" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:06.248688 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.248669    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-podinfo" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:06.248771 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.248699    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-token" pod "e8db72a9-39e5-4274-aff2-856577cfe58d" (UID: "e8db72a9-39e5-4274-aff2-856577cfe58d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:06.348480 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.348445    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.348480 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.348481    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.348695 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.348495    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-967wz\" (UniqueName: \"kubernetes.io/projected/e8db72a9-39e5-4274-aff2-856577cfe58d-kube-api-access-967wz\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.348695 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.348510    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8db72a9-39e5-4274-aff2-856577cfe58d-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:06.553682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.553648    5002 generic.go:356] "Generic (PLEG): container finished" podID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerID="c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79" exitCode=0
Oct 30 20:47:06.553899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.553715    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"
Oct 30 20:47:06.553899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.553738    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" event={"ID":"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d","Type":"ContainerDied","Data":"c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79"}
Oct 30 20:47:06.553899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.553814    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v" event={"ID":"c88c8a7e-8ab5-466b-abfb-2ca8de7a914d","Type":"ContainerDied","Data":"6d39b0264a0f3e709c76edd75297ecc01861440b07a9eae9e308fa1c4be63867"}
Oct 30 20:47:06.553899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.553839    5002 scope.go:122] "RemoveContainer" containerID="c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79"
Oct 30 20:47:06.555783 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.555757    5002 generic.go:356] "Generic (PLEG): container finished" podID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerID="661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f" exitCode=0
Oct 30 20:47:06.555916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.555847    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"
Oct 30 20:47:06.555916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.555882    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" event={"ID":"e8db72a9-39e5-4274-aff2-856577cfe58d","Type":"ContainerDied","Data":"661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f"}
Oct 30 20:47:06.556046 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.555917    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89" event={"ID":"e8db72a9-39e5-4274-aff2-856577cfe58d","Type":"ContainerDied","Data":"60316900c515c4dbe23dcc4c3fa56159dce21e8720529a447f1aa16c6e15ee50"}
Oct 30 20:47:06.564940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.564923    5002 scope.go:122] "RemoveContainer" containerID="9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564"
Oct 30 20:47:06.575095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.575077    5002 scope.go:122] "RemoveContainer" containerID="c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79"
Oct 30 20:47:06.575360 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:06.575332    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79\": container with ID starting with c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79 not found: ID does not exist" containerID="c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79"
Oct 30 20:47:06.575415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.575365    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79"} err="failed to get container status \"c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79\": rpc error: code = NotFound desc = could not find container \"c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79\": container with ID starting with c0dfd4a799d7ae1aca2315b4add33318f58cd398057a34b6fbc3235e60581b79 not found: ID does not exist"
Oct 30 20:47:06.575415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.575388    5002 scope.go:122] "RemoveContainer" containerID="9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564"
Oct 30 20:47:06.575685 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:06.575664    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564\": container with ID starting with 9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564 not found: ID does not exist" containerID="9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564"
Oct 30 20:47:06.575753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.575691    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564"} err="failed to get container status \"9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564\": rpc error: code = NotFound desc = could not find container \"9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564\": container with ID starting with 9c6290328f0ed1c3621ab02974ed3712d0d67cba932166ca54759f6f9368b564 not found: ID does not exist"
Oct 30 20:47:06.575753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.575705    5002 scope.go:122] "RemoveContainer" containerID="661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f"
Oct 30 20:47:06.584727 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.584709    5002 scope.go:122] "RemoveContainer" containerID="85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849"
Oct 30 20:47:06.593906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.593885    5002 scope.go:122] "RemoveContainer" containerID="661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f"
Oct 30 20:47:06.594161 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:06.594142    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f\": container with ID starting with 661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f not found: ID does not exist" containerID="661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f"
Oct 30 20:47:06.594235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.594170    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f"} err="failed to get container status \"661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f\": rpc error: code = NotFound desc = could not find container \"661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f\": container with ID starting with 661c1560bdc7c3e404185f5042b9b97cc53140e40fbc780b46cba27d3e63424f not found: ID does not exist"
Oct 30 20:47:06.594235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.594189    5002 scope.go:122] "RemoveContainer" containerID="85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849"
Oct 30 20:47:06.594443 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:06.594426    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849\": container with ID starting with 85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849 not found: ID does not exist" containerID="85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849"
Oct 30 20:47:06.594496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.594451    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849"} err="failed to get container status \"85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849\": rpc error: code = NotFound desc = could not find container \"85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849\": container with ID starting with 85ae18d1b0a2d264e5657b729c2f523be060a3bddbafa7edf5f8e0c27264e849 not found: ID does not exist"
Oct 30 20:47:06.645223 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.645197    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:06.728581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.728552    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"]
Oct 30 20:47:06.779109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.779086    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v"]
Oct 30 20:47:06.849349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.849291    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"]
Oct 30 20:47:06.852582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.852558    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:06.856055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:06.856035    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89"]
Oct 30 20:47:07.048323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:07.048301    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:07.117058 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:07.116989    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:47:07.329993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:07.329959    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:07.574154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:07.574113    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:07.917744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:07.917667    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:07.968678 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:07.968648    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" path="/var/lib/kubelet/pods/c88c8a7e-8ab5-466b-abfb-2ca8de7a914d/volumes"
Oct 30 20:47:07.969135 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:07.969121    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" path="/var/lib/kubelet/pods/e8db72a9-39e5-4274-aff2-856577cfe58d/volumes"
Oct 30 20:47:09.593325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:09.593292    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"]
Oct 30 20:47:09.593833 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:09.593496    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy" containerID="cri-o://29dcdd9f0baabf0b3dfc7659d386573f69d554376d18f3ae74be4657a9c6fe56" gracePeriod=30
Oct 30 20:47:09.975015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:09.974984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:10.182807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.182764    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:10.498040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.498001    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"]
Oct 30 20:47:10.499051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499030    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy"
Oct 30 20:47:10.499123 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499055    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy"
Oct 30 20:47:10.499123 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499074    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy"
Oct 30 20:47:10.499123 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499082    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy"
Oct 30 20:47:10.499123 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499118    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy"
Oct 30 20:47:10.499242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499127    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy"
Oct 30 20:47:10.499361 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499348    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy"
Oct 30 20:47:10.499396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499378    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy"
Oct 30 20:47:10.499396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.499389    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88c8a7e-8ab5-466b-abfb-2ca8de7a914d" containerName="istio-proxy"
Oct 30 20:47:10.505677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.505655    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.561826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.559969    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-0ac4e5\""
Oct 30 20:47:10.573555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.570796    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-0ac4e5-dockercfg-5ps56\""
Oct 30 20:47:10.637561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.634890    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"]
Oct 30 20:47:10.705663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.701630    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"]
Oct 30 20:47:10.707515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.707483    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aad480-43f9-45dd-8f51-2924f7091faf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.707842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.707818    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlpp\" (UniqueName: \"kubernetes.io/projected/62aad480-43f9-45dd-8f51-2924f7091faf-kube-api-access-mtlpp\") pod \"payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.708703 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.708681    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy"
Oct 30 20:47:10.708829 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.708815    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy"
Oct 30 20:47:10.709102 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.709085    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8db72a9-39e5-4274-aff2-856577cfe58d" containerName="istio-proxy"
Oct 30 20:47:10.715166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.715147    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:10.732481 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.732455    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"]
Oct 30 20:47:10.808902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.808823    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlpp\" (UniqueName: \"kubernetes.io/projected/62aad480-43f9-45dd-8f51-2924f7091faf-kube-api-access-mtlpp\") pod \"payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.809012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.808964    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3ed774-c246-488d-828b-0bb4126a9fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:10.809012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.809007    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aad480-43f9-45dd-8f51-2924f7091faf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.809124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.809104    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnt5v\" (UniqueName: \"kubernetes.io/projected/1e3ed774-c246-488d-828b-0bb4126a9fb0-kube-api-access-gnt5v\") pod \"payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:10.809661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.809642    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aad480-43f9-45dd-8f51-2924f7091faf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.866738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.866711    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlpp\" (UniqueName: \"kubernetes.io/projected/62aad480-43f9-45dd-8f51-2924f7091faf-kube-api-access-mtlpp\") pod \"payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.910050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.910024    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3ed774-c246-488d-828b-0bb4126a9fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:10.910144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.910107    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnt5v\" (UniqueName: \"kubernetes.io/projected/1e3ed774-c246-488d-828b-0bb4126a9fb0-kube-api-access-gnt5v\") pod \"payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:10.910659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.910638    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3ed774-c246-488d-828b-0bb4126a9fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:10.918241 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.918218    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:10.968762 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:10.968733    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnt5v\" (UniqueName: \"kubernetes.io/projected/1e3ed774-c246-488d-828b-0bb4126a9fb0-kube-api-access-gnt5v\") pod \"payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:11.039301 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.033703    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:11.202062 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.202028    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"]
Oct 30 20:47:11.208722 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.208700    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.216129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.216102    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwm9\" (UniqueName: \"kubernetes.io/projected/2145966a-2cd2-458b-a7db-5020d14a4913-kube-api-access-7pwm9\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.216302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.216276    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2145966a-2cd2-458b-a7db-5020d14a4913-openshift-service-ca\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.216377 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.216318    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-metrics-certs\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.216515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.216493    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.261840 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.261805    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-27219c73-serving-cert\""
Oct 30 20:47:11.263860 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.263836    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"]
Oct 30 20:47:11.315716 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:11.315670    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62aad480_43f9_45dd_8f51_2924f7091faf.slice/crio-634c0eb0aca35b559ab3e70a4c731ddd8b4a9cb824cd31fec0a055f7fb2e7744 WatchSource:0}: Error finding container 634c0eb0aca35b559ab3e70a4c731ddd8b4a9cb824cd31fec0a055f7fb2e7744: Status 404 returned error can't find the container with id 634c0eb0aca35b559ab3e70a4c731ddd8b4a9cb824cd31fec0a055f7fb2e7744
Oct 30 20:47:11.317038 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.317014    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"]
Oct 30 20:47:11.317599 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.317514    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2145966a-2cd2-458b-a7db-5020d14a4913-openshift-service-ca\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.317691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.317611    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-metrics-certs\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.317750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.317695    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.317836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.317820    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm9\" (UniqueName: \"kubernetes.io/projected/2145966a-2cd2-458b-a7db-5020d14a4913-kube-api-access-7pwm9\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.318132 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:11.318112    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-serving-cert: secret "maas-api-e2e-mt-27219c73-serving-cert" not found
Oct 30 20:47:11.318227 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:11.318175    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls podName:2145966a-2cd2-458b-a7db-5020d14a4913 nodeName:}" failed. No retries permitted until 2026-10-30 20:47:11.818155142 +0000 UTC m=+1657.967206125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls") pod "maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s" (UID: "2145966a-2cd2-458b-a7db-5020d14a4913") : secret "maas-api-e2e-mt-27219c73-serving-cert" not found
Oct 30 20:47:11.318271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.318230    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2145966a-2cd2-458b-a7db-5020d14a4913-openshift-service-ca\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.320374 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.320348    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-metrics-certs\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.396715 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.377578    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwm9\" (UniqueName: \"kubernetes.io/projected/2145966a-2cd2-458b-a7db-5020d14a4913-kube-api-access-7pwm9\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.429751 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:11.429725    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3ed774_c246_488d_828b_0bb4126a9fb0.slice/crio-67f79b2290e01eb2f4a8ef2b2307b1d7650440cb991c966689561087c20b94bc WatchSource:0}: Error finding container 67f79b2290e01eb2f4a8ef2b2307b1d7650440cb991c966689561087c20b94bc: Status 404 returned error can't find the container with id 67f79b2290e01eb2f4a8ef2b2307b1d7650440cb991c966689561087c20b94bc
Oct 30 20:47:11.431324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.431301    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"]
Oct 30 20:47:11.465794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.465758    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:11.677100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.677020    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:11.695221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.695194    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66" event={"ID":"62aad480-43f9-45dd-8f51-2924f7091faf","Type":"ContainerStarted","Data":"350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a"}
Oct 30 20:47:11.695391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.695249    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66" event={"ID":"62aad480-43f9-45dd-8f51-2924f7091faf","Type":"ContainerStarted","Data":"634c0eb0aca35b559ab3e70a4c731ddd8b4a9cb824cd31fec0a055f7fb2e7744"}
Oct 30 20:47:11.695391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.695279    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:11.696949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.696921    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp" event={"ID":"1e3ed774-c246-488d-828b-0bb4126a9fb0","Type":"ContainerStarted","Data":"b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5"}
Oct 30 20:47:11.697047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.696974    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp" event={"ID":"1e3ed774-c246-488d-828b-0bb4126a9fb0","Type":"ContainerStarted","Data":"67f79b2290e01eb2f4a8ef2b2307b1d7650440cb991c966689561087c20b94bc"}
Oct 30 20:47:11.697047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.697000    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:11.820387 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.820334    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66" podStartSLOduration=1.820317103 podStartE2EDuration="1.820317103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:11.818672182 +0000 UTC m=+1657.967723168" watchObservedRunningTime="2026-10-30 20:47:11.820317103 +0000 UTC m=+1657.969368089"
Oct 30 20:47:11.831966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.831939    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.834982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.834954    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:11.955917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.955791    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp" podStartSLOduration=1.95577384 podStartE2EDuration="1.95577384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:11.942700568 +0000 UTC m=+1658.091751572" watchObservedRunningTime="2026-10-30 20:47:11.95577384 +0000 UTC m=+1658.104824827"
Oct 30 20:47:11.960341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:11.960290    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:12.125916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:12.125338    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:12.522638 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:12.522600    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2145966a_2cd2_458b_a7db_5020d14a4913.slice/crio-cf4963e94cd337a4885438f9ba04746641cd289cf7dd6b848c5aa8eb17b0df39 WatchSource:0}: Error finding container cf4963e94cd337a4885438f9ba04746641cd289cf7dd6b848c5aa8eb17b0df39: Status 404 returned error can't find the container with id cf4963e94cd337a4885438f9ba04746641cd289cf7dd6b848c5aa8eb17b0df39
Oct 30 20:47:12.523097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:12.523077    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"]
Oct 30 20:47:12.524049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:12.524032    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:47:12.704809 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:12.704776    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s" event={"ID":"2145966a-2cd2-458b-a7db-5020d14a4913","Type":"ContainerStarted","Data":"cf4963e94cd337a4885438f9ba04746641cd289cf7dd6b848c5aa8eb17b0df39"}
Oct 30 20:47:12.847572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:12.847480    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"]
Oct 30 20:47:12.847725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:12.847701    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy" containerID="cri-o://2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861" gracePeriod=30
Oct 30 20:47:13.714341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:13.714301    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s" event={"ID":"2145966a-2cd2-458b-a7db-5020d14a4913","Type":"ContainerStarted","Data":"ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2"}
Oct 30 20:47:13.714817 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:13.714367    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:13.836341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:13.836240    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s" podStartSLOduration=1.880157637 podStartE2EDuration="2.836223943s" totalImagesPullingTime="956.066306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:13.832763645 +0000 UTC m=+1659.981814632" watchObservedRunningTime="2026-10-30 20:47:13.836223943 +0000 UTC m=+1659.985274931"
Oct 30 20:47:14.735600 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.735572    5002 generic.go:356] "Generic (PLEG): container finished" podID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerID="29dcdd9f0baabf0b3dfc7659d386573f69d554376d18f3ae74be4657a9c6fe56" exitCode=0
Oct 30 20:47:14.736053 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.735660    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" event={"ID":"4dda145c-244b-4d9b-9888-06ced80ac5ef","Type":"ContainerDied","Data":"29dcdd9f0baabf0b3dfc7659d386573f69d554376d18f3ae74be4657a9c6fe56"}
Oct 30 20:47:14.910065 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.910002    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:47:14.991614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991591    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-envoy\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.991728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991646    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-podinfo\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.991728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991689    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dda145c-244b-4d9b-9888-06ced80ac5ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda145c-244b-4d9b-9888-06ced80ac5ef-istiod-ca-cert\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.991836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991760    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-data\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.991836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991808    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-kube-api-access-65lh4\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-kube-api-access-65lh4\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.991937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991840    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-certs\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.991937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991889    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-socket\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.991937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991912    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-credential-socket\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.992096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.991947    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-token\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-token\") pod \"4dda145c-244b-4d9b-9888-06ced80ac5ef\" (UID: \"4dda145c-244b-4d9b-9888-06ced80ac5ef\") "
Oct 30 20:47:14.992096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992079    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-data" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:14.992200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992096    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dda145c-244b-4d9b-9888-06ced80ac5ef-istiod-ca-cert" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:14.992328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992303    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-credential-socket" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:14.992457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992438    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-socket" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:14.992457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992441    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-certs" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:14.992787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992712    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:14.992904 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992860    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:14.992942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992909    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda145c-244b-4d9b-9888-06ced80ac5ef-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:14.992942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992923    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:14.992942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.992936    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:14.994060 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.994034    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-podinfo" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:14.994326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.994307    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-token" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:14.994413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.994393    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-envoy" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:14.994455 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:14.994424    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-kube-api-access-65lh4" pod "4dda145c-244b-4d9b-9888-06ced80ac5ef" (UID: "4dda145c-244b-4d9b-9888-06ced80ac5ef"). InnerVolumeSpecName "kube-api-access-65lh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:15.094664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.094641    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65lh4\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-kube-api-access-65lh4\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:15.094664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.094662    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:15.094803 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.094672    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:15.094803 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.094681    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda145c-244b-4d9b-9888-06ced80ac5ef-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:15.744213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.744182    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"
Oct 30 20:47:15.744647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.744213    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j" event={"ID":"4dda145c-244b-4d9b-9888-06ced80ac5ef","Type":"ContainerDied","Data":"705d39bb7b0882ad35988a651c878e6ef52c3fcdf3a537240a012d993c27421e"}
Oct 30 20:47:15.744647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.744278    5002 scope.go:122] "RemoveContainer" containerID="29dcdd9f0baabf0b3dfc7659d386573f69d554376d18f3ae74be4657a9c6fe56"
Oct 30 20:47:15.979737 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:15.979704    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"]
Oct 30 20:47:16.020224 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:16.020154    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j"]
Oct 30 20:47:16.837305 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:16.837271    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:17.613438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.613389    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:17.793464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.793432    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:17.969333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.969306    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" path="/var/lib/kubelet/pods/4dda145c-244b-4d9b-9888-06ced80ac5ef/volumes"
Oct 30 20:47:17.976939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.976916    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"]
Oct 30 20:47:17.977577 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.977562    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy"
Oct 30 20:47:17.977640 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.977578    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy"
Oct 30 20:47:17.977742 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.977732    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dda145c-244b-4d9b-9888-06ced80ac5ef" containerName="istio-proxy"
Oct 30 20:47:17.981520 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:17.981435    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.054788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.054756    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"]
Oct 30 20:47:18.155611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.148264    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-99c030-dockercfg-rbbz2\""
Oct 30 20:47:18.155611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.148363    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-99c030\""
Oct 30 20:47:18.286859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.286801    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"]
Oct 30 20:47:18.296589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.292789    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8lx\" (UniqueName: \"kubernetes.io/projected/cd640b3a-cea1-499f-a2fb-5e118f46407d-kube-api-access-tr8lx\") pod \"payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.296589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.292845    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd640b3a-cea1-499f-a2fb-5e118f46407d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.300876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.300850    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"]
Oct 30 20:47:18.301010 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.300966    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.312157 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.312129    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:47:18.394326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.394297    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1487eb-98e0-4f47-bd93-95ab47d1f626-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.394474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.394450    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8lx\" (UniqueName: \"kubernetes.io/projected/cd640b3a-cea1-499f-a2fb-5e118f46407d-kube-api-access-tr8lx\") pod \"payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.394562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.394509    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd640b3a-cea1-499f-a2fb-5e118f46407d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.394693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.394675    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wr2z\" (UniqueName: \"kubernetes.io/projected/5b1487eb-98e0-4f47-bd93-95ab47d1f626-kube-api-access-8wr2z\") pod \"payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.395138 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.395115    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd640b3a-cea1-499f-a2fb-5e118f46407d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.452575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.452513    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8lx\" (UniqueName: \"kubernetes.io/projected/cd640b3a-cea1-499f-a2fb-5e118f46407d-kube-api-access-tr8lx\") pod \"payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.495666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495644    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-data\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.495772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495696    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-credential-socket\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.495772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495729    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-podinfo\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.495868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495782    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-certs\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.495868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495817    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e7cdb7-12ab-462a-9c63-b18e2038e659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e7cdb7-12ab-462a-9c63-b18e2038e659-istiod-ca-cert\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.495868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495838    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-token\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-token\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.496002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495876    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-envoy\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.496002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495876    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-data" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:18.496002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495944    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-socket\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.496002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.495977    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-credential-socket" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:18.496002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496000    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-kube-api-access-bklns\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-kube-api-access-bklns\") pod \"55e7cdb7-12ab-462a-9c63-b18e2038e659\" (UID: \"55e7cdb7-12ab-462a-9c63-b18e2038e659\") "
Oct 30 20:47:18.496252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496149    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-socket" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:18.496309 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496249    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e7cdb7-12ab-462a-9c63-b18e2038e659-istiod-ca-cert" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:18.496437 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496413    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wr2z\" (UniqueName: \"kubernetes.io/projected/5b1487eb-98e0-4f47-bd93-95ab47d1f626-kube-api-access-8wr2z\") pod \"payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.496843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496510    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-certs" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:18.496843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496595    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1487eb-98e0-4f47-bd93-95ab47d1f626-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.496843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496821    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.496843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496842    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.497271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496858    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.497271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496872    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e7cdb7-12ab-462a-9c63-b18e2038e659-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.497271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.496894    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.497271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.497252    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1487eb-98e0-4f47-bd93-95ab47d1f626-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.497957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.497932    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-podinfo" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:18.498515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.498496    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-token" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:18.498607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.498588    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-envoy" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:18.498749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.498730    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-kube-api-access-bklns" pod "55e7cdb7-12ab-462a-9c63-b18e2038e659" (UID: "55e7cdb7-12ab-462a-9c63-b18e2038e659"). InnerVolumeSpecName "kube-api-access-bklns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:18.555144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.555114    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wr2z\" (UniqueName: \"kubernetes.io/projected/5b1487eb-98e0-4f47-bd93-95ab47d1f626-kube-api-access-8wr2z\") pod \"payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.597827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.597801    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklns\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-kube-api-access-bklns\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.597827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.597825    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.597969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.597836    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.597969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.597844    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e7cdb7-12ab-462a-9c63-b18e2038e659-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.608260 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.608232    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:18.625940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.625911    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:18.789760 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.789724    5002 generic.go:356] "Generic (PLEG): container finished" podID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerID="2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861" exitCode=0
Oct 30 20:47:18.789916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.789798    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" event={"ID":"55e7cdb7-12ab-462a-9c63-b18e2038e659","Type":"ContainerDied","Data":"2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861"}
Oct 30 20:47:18.789916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.789802    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"
Oct 30 20:47:18.789916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.789850    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g" event={"ID":"55e7cdb7-12ab-462a-9c63-b18e2038e659","Type":"ContainerDied","Data":"ff6ae7025ea4a6d083f9a1fb1d4de49b8dfa2bbacff76f74bc923864c89bba02"}
Oct 30 20:47:18.789916 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.789877    5002 scope.go:122] "RemoveContainer" containerID="2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861"
Oct 30 20:47:18.801886 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.801688    5002 scope.go:122] "RemoveContainer" containerID="2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861"
Oct 30 20:47:18.802038 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:18.802015    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861\": container with ID starting with 2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861 not found: ID does not exist" containerID="2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861"
Oct 30 20:47:18.802117 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.802060    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861"} err="failed to get container status \"2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861\": rpc error: code = NotFound desc = could not find container \"2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861\": container with ID starting with 2b79e82b61c3251abbb46bbdfe824bfa12faf75981f9c51523b11e2ee753f861 not found: ID does not exist"
Oct 30 20:47:18.970871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.970845    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"]
Oct 30 20:47:18.974690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:18.974670    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g"]
Oct 30 20:47:19.006747 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.006726    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"]
Oct 30 20:47:19.035728 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:19.035702    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd640b3a_cea1_499f_a2fb_5e118f46407d.slice/crio-ba76b4e52eeb592bf187e73f23c712a972ebfcf361727c34e67ab4e69331149b WatchSource:0}: Error finding container ba76b4e52eeb592bf187e73f23c712a972ebfcf361727c34e67ab4e69331149b: Status 404 returned error can't find the container with id ba76b4e52eeb592bf187e73f23c712a972ebfcf361727c34e67ab4e69331149b
Oct 30 20:47:19.059330 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.059306    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"]
Oct 30 20:47:19.059420 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:19.059330    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1487eb_98e0_4f47_bd93_95ab47d1f626.slice/crio-e40a9cc9d368eb9a49f6caeaf933d3bace620750e135c2672ee2576d7b6ffd41 WatchSource:0}: Error finding container e40a9cc9d368eb9a49f6caeaf933d3bace620750e135c2672ee2576d7b6ffd41: Status 404 returned error can't find the container with id e40a9cc9d368eb9a49f6caeaf933d3bace620750e135c2672ee2576d7b6ffd41
Oct 30 20:47:19.741509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.741482    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:47:19.799041 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.799014    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4" event={"ID":"cd640b3a-cea1-499f-a2fb-5e118f46407d","Type":"ContainerStarted","Data":"7f4638d469dcfabe8a1fd0635d8aca55a59185a16e97396d2dda3353225b555b"}
Oct 30 20:47:19.799201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.799058    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4" event={"ID":"cd640b3a-cea1-499f-a2fb-5e118f46407d","Type":"ContainerStarted","Data":"ba76b4e52eeb592bf187e73f23c712a972ebfcf361727c34e67ab4e69331149b"}
Oct 30 20:47:19.799201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.799077    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:19.801807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.801782    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl" event={"ID":"5b1487eb-98e0-4f47-bd93-95ab47d1f626","Type":"ContainerStarted","Data":"34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85"}
Oct 30 20:47:19.801956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.801820    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl" event={"ID":"5b1487eb-98e0-4f47-bd93-95ab47d1f626","Type":"ContainerStarted","Data":"e40a9cc9d368eb9a49f6caeaf933d3bace620750e135c2672ee2576d7b6ffd41"}
Oct 30 20:47:19.801956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.801858    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:19.968895 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.968854    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" path="/var/lib/kubelet/pods/55e7cdb7-12ab-462a-9c63-b18e2038e659/volumes"
Oct 30 20:47:19.981217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:19.981170    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl" podStartSLOduration=1.981157269 podStartE2EDuration="1.981157269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:19.978331169 +0000 UTC m=+1666.127382156" watchObservedRunningTime="2026-10-30 20:47:19.981157269 +0000 UTC m=+1666.130208254"
Oct 30 20:47:20.101172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:20.101065    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4" podStartSLOduration=3.101047863 podStartE2EDuration="3.101047863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:20.094454866 +0000 UTC m=+1666.243505854" watchObservedRunningTime="2026-10-30 20:47:20.101047863 +0000 UTC m=+1666.250098851"
Oct 30 20:47:21.644224 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:21.644185    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:22.708637 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:22.708611    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:47:22.709055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:22.708682    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:47:26.331142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:26.331005    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:27.956104 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:27.956072    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"]
Oct 30 20:47:27.956590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:27.956288    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" containerID="cri-o://5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f" gracePeriod=30
Oct 30 20:47:28.005061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.005034    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"]
Oct 30 20:47:28.005235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.005208    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy" containerID="cri-o://bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047" gracePeriod=30
Oct 30 20:47:28.009363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.009340    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"]
Oct 30 20:47:28.009509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.009492    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" containerID="cri-o://134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7" gracePeriod=30
Oct 30 20:47:28.056630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.056603    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"]
Oct 30 20:47:28.056844 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.056817    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy" containerID="cri-o://d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef" gracePeriod=30
Oct 30 20:47:28.106905 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.106861    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"]
Oct 30 20:47:28.107081 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:28.107034    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy" containerID="cri-o://4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e" gracePeriod=30
Oct 30 20:47:30.815369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:30.815338    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:47:30.815866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:30.815428    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:47:31.171089 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:31.171011    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:47:31.171254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:31.171081    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:47:31.375064 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:31.375031    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:32.769887 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:32.769851    5002 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:47:32.770288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:32.769929    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:47:33.312307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.309572    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:47:33.375568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.375498    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aacf197-1de8-403b-8e73-c3cfd73452a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aacf197-1de8-403b-8e73-c3cfd73452a0-istiod-ca-cert\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.375716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.375580    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-podinfo\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.375716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.375650    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-credential-socket\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.375833 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.375748    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-certs\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.375884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.375873    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-data\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.375935 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.375922    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-socket\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.376090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.376065    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-kube-api-access-hgqf4\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-kube-api-access-hgqf4\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.376151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.376112    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-token\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-token\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.376151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.376142    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-envoy\") pod \"1aacf197-1de8-403b-8e73-c3cfd73452a0\" (UID: \"1aacf197-1de8-403b-8e73-c3cfd73452a0\") "
Oct 30 20:47:33.377663 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.377637    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-certs" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.378771 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.378702    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aacf197-1de8-403b-8e73-c3cfd73452a0-istiod-ca-cert" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:33.380576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.380266    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-data" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.380576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.380331    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-credential-socket" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.380576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.380412    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-socket" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.381170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.381135    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-podinfo" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:33.382647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.382619    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-kube-api-access-hgqf4" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "kube-api-access-hgqf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.382714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.382693    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-token" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.383134 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.383114    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-envoy" pod "1aacf197-1de8-403b-8e73-c3cfd73452a0" (UID: "1aacf197-1de8-403b-8e73-c3cfd73452a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.412461 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.412385    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:47:33.468834 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.468096    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:47:33.477186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477162    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-token\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-token\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.477323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477228    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-data\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.477323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477259    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-envoy\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.477323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477290    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-envoy\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.477483 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477334    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-credential-socket\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.477483 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477371    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-podinfo\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.477483 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477454    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-kube-api-access-bscmx\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-kube-api-access-bscmx\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.477671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477487    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-token\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.477671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477518    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-socket\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.477671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477629    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-podinfo\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.477814 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477750    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-certs\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.477814 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477780    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istiod-ca-cert\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.477912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477850    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-certs\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.477912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477906    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-data\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.478109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477968    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istiod-ca-cert\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.478109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.477997    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-kube-api-access-xqwfr\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-kube-api-access-xqwfr\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.478109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.478050    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-credential-socket\") pod \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\" (UID: \"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487\") "
Oct 30 20:47:33.478334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.478131    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-socket\") pod \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\" (UID: \"56204e2f-0989-4a1a-a7c2-dd9a707b827e\") "
Oct 30 20:47:33.478870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.478664    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-socket" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.478956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.478900    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-socket" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479164    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479193    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgqf4\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-kube-api-access-hgqf4\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479215    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479231    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479244    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aacf197-1de8-403b-8e73-c3cfd73452a0-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479257    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479272    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479286    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479299    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479314    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.479765 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479326    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aacf197-1de8-403b-8e73-c3cfd73452a0-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.480385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.479958    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-certs" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.480385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.480242    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istiod-ca-cert" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:33.480553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.480512    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-certs" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.481364 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.481342    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-podinfo" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:33.482250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.481979    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-data" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.482250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.482205    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-credential-socket" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.482417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.482404    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istiod-ca-cert" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:33.482479 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.482413    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-data" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.482479 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.482452    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-credential-socket" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.486659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.486635    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-envoy" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.487332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.487312    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-podinfo" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:33.487695 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.487651    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-envoy" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.487794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.487770    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-token" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.488033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.488001    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-kube-api-access-xqwfr" pod "56204e2f-0989-4a1a-a7c2-dd9a707b827e" (UID: "56204e2f-0989-4a1a-a7c2-dd9a707b827e"). InnerVolumeSpecName "kube-api-access-xqwfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.488128 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.488105    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-token" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.488252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.488234    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-kube-api-access-bscmx" pod "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" (UID: "ce1f033c-7faa-4d52-8c3a-f8e50e5c0487"). InnerVolumeSpecName "kube-api-access-bscmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.510771 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.510704    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:47:33.524962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.524942    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:47:33.547127 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.547064    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:33.580664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.580635    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istiod-ca-cert\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.580785 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.580684    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-kube-api-access-wrfdw\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-kube-api-access-wrfdw\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.580785 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.580713    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-podinfo\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.580785 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.580758    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-socket\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.580900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.580782    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-token\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.580900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.580827    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-certs\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.581184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581055    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istiod-ca-cert" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:33.581459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581434    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-socket" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.581559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581442    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-certs" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.581559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581511    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-certs\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.581810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581579    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-data\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.581810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581647    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-credential-socket\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.581810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581747    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-certs" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.581978 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581946    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-credential-socket" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.582133 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582089    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-data" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.582133 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.581884    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-data" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.582294 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582228    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-data\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.582358 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582302    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-socket\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.582358 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582333    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942ad519-417a-4b3f-be9c-5e104267f692-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942ad519-417a-4b3f-be9c-5e104267f692-istiod-ca-cert\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.582455 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582362    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-istio-token\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-istio-token\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.582510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582466    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-envoy\") pod \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\" (UID: \"0e4eca8a-ec71-4ee5-b142-91786d5364e7\") "
Oct 30 20:47:33.582581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582515    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-envoy\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.583385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582610    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/942ad519-417a-4b3f-be9c-5e104267f692-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942ad519-417a-4b3f-be9c-5e104267f692-istio-podinfo\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.583385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582643    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-kube-api-access-jdbxt\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-kube-api-access-jdbxt\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.583385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582696    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-socket" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.583385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.582733    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-credential-socket\") pod \"942ad519-417a-4b3f-be9c-5e104267f692\" (UID: \"942ad519-417a-4b3f-be9c-5e104267f692\") "
Oct 30 20:47:33.583385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583019    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942ad519-417a-4b3f-be9c-5e104267f692-istiod-ca-cert" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:33.583385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583376    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583396    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583411    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583426    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583443    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583457    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583474    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583486    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583501    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583516    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscmx\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-kube-api-access-bscmx\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583549    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583563    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583576    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583590    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583604    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583618    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583631    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583645    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942ad519-417a-4b3f-be9c-5e104267f692-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583658    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56204e2f-0989-4a1a-a7c2-dd9a707b827e-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583671    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56204e2f-0989-4a1a-a7c2-dd9a707b827e-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583684    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583697    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583710    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583726    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwfr\" (UniqueName: \"kubernetes.io/projected/56204e2f-0989-4a1a-a7c2-dd9a707b827e-kube-api-access-xqwfr\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.583910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.583739    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.585425 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.584291    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-token" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.585425 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.584878    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-credential-socket" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.585425 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.585410    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-kube-api-access-wrfdw" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "kube-api-access-wrfdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.585761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.585737    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-podinfo" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:33.586343 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.586317    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/942ad519-417a-4b3f-be9c-5e104267f692-istio-podinfo" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:33.586434 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.586343    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-kube-api-access-jdbxt" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "kube-api-access-jdbxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.586522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.586496    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-istio-token" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.587936 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.587912    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-envoy" pod "942ad519-417a-4b3f-be9c-5e104267f692" (UID: "942ad519-417a-4b3f-be9c-5e104267f692"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.588054 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.588037    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-envoy" pod "0e4eca8a-ec71-4ee5-b142-91786d5364e7" (UID: "0e4eca8a-ec71-4ee5-b142-91786d5364e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.685184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685155    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685183    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685193    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942ad519-417a-4b3f-be9c-5e104267f692-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685202    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbxt\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-kube-api-access-jdbxt\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685212    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942ad519-417a-4b3f-be9c-5e104267f692-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685221    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrfdw\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-kube-api-access-wrfdw\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685229    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685237    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eca8a-ec71-4ee5-b142-91786d5364e7-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.685390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:33.685245    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942ad519-417a-4b3f-be9c-5e104267f692-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:34.021150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.021121    5002 generic.go:356] "Generic (PLEG): container finished" podID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerID="5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f" exitCode=0
Oct 30 20:47:34.021627 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.021222    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" event={"ID":"1aacf197-1de8-403b-8e73-c3cfd73452a0","Type":"ContainerDied","Data":"5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f"}
Oct 30 20:47:34.021627 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.021259    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd" event={"ID":"1aacf197-1de8-403b-8e73-c3cfd73452a0","Type":"ContainerDied","Data":"d10b831a398e5dd8efbbb3d08c5aab2e9bc764e94afb89a8028d84516775b49d"}
Oct 30 20:47:34.021627 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.021284    5002 scope.go:122] "RemoveContainer" containerID="5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f"
Oct 30 20:47:34.021627 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.021402    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"
Oct 30 20:47:34.024331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.024303    5002 generic.go:356] "Generic (PLEG): container finished" podID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerID="134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7" exitCode=0
Oct 30 20:47:34.024434 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.024420    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" event={"ID":"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487","Type":"ContainerDied","Data":"134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7"}
Oct 30 20:47:34.024498 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.024460    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh" event={"ID":"ce1f033c-7faa-4d52-8c3a-f8e50e5c0487","Type":"ContainerDied","Data":"a61b44dbed661ad7b1b642d74eb2b18a5d0a049d92738a6450cfa8ce98bbc18a"}
Oct 30 20:47:34.024937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.024912    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"
Oct 30 20:47:34.026020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.025993    5002 generic.go:356] "Generic (PLEG): container finished" podID="942ad519-417a-4b3f-be9c-5e104267f692" containerID="d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef" exitCode=0
Oct 30 20:47:34.026163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.026148    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"
Oct 30 20:47:34.027146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.027121    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" event={"ID":"942ad519-417a-4b3f-be9c-5e104267f692","Type":"ContainerDied","Data":"d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef"}
Oct 30 20:47:34.027228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.027172    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf" event={"ID":"942ad519-417a-4b3f-be9c-5e104267f692","Type":"ContainerDied","Data":"a4b9bab96ebd84f91d46f6b66b4db1a2f638eb21ce17584b778bf3f48bdc1eae"}
Oct 30 20:47:34.029397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.029374    5002 generic.go:356] "Generic (PLEG): container finished" podID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerID="bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047" exitCode=0
Oct 30 20:47:34.029476 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.029433    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"
Oct 30 20:47:34.029616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.029598    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" event={"ID":"56204e2f-0989-4a1a-a7c2-dd9a707b827e","Type":"ContainerDied","Data":"bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047"}
Oct 30 20:47:34.029673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.029642    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn" event={"ID":"56204e2f-0989-4a1a-a7c2-dd9a707b827e","Type":"ContainerDied","Data":"1870ef46952b44c09ce12600913ba90b3bd10f19c0d55c8fab632fae93720c5c"}
Oct 30 20:47:34.033352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.033315    5002 generic.go:356] "Generic (PLEG): container finished" podID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerID="4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e" exitCode=0
Oct 30 20:47:34.035313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.033419    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"
Oct 30 20:47:34.035313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.033546    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" event={"ID":"0e4eca8a-ec71-4ee5-b142-91786d5364e7","Type":"ContainerDied","Data":"4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e"}
Oct 30 20:47:34.035313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.033620    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs" event={"ID":"0e4eca8a-ec71-4ee5-b142-91786d5364e7","Type":"ContainerDied","Data":"f754bcbe28ceb761a9822e389a961f34443db1bbef0fbb4ed94b4b4a99f371c9"}
Oct 30 20:47:34.045095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.044847    5002 scope.go:122] "RemoveContainer" containerID="5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f"
Oct 30 20:47:34.045610 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:34.045583    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f\": container with ID starting with 5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f not found: ID does not exist" containerID="5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f"
Oct 30 20:47:34.045685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.045631    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f"} err="failed to get container status \"5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f\": rpc error: code = NotFound desc = could not find container \"5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f\": container with ID starting with 5e8a69bff478900b77657d96c6a626b2730d9501dce004198e394cd5d4daeb9f not found: ID does not exist"
Oct 30 20:47:34.045685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.045654    5002 scope.go:122] "RemoveContainer" containerID="134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7"
Oct 30 20:47:34.060796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.060780    5002 scope.go:122] "RemoveContainer" containerID="134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7"
Oct 30 20:47:34.061078 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:34.061052    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7\": container with ID starting with 134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7 not found: ID does not exist" containerID="134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7"
Oct 30 20:47:34.061160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.061086    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7"} err="failed to get container status \"134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7\": rpc error: code = NotFound desc = could not find container \"134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7\": container with ID starting with 134c85e3e3f4c6ded0115dea352fa48da0f4ae88e71a27a4bf5929224a8328f7 not found: ID does not exist"
Oct 30 20:47:34.061160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.061102    5002 scope.go:122] "RemoveContainer" containerID="d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef"
Oct 30 20:47:34.073008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.072981    5002 scope.go:122] "RemoveContainer" containerID="d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef"
Oct 30 20:47:34.073393 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:34.073331    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef\": container with ID starting with d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef not found: ID does not exist" containerID="d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef"
Oct 30 20:47:34.073393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.073367    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef"} err="failed to get container status \"d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef\": rpc error: code = NotFound desc = could not find container \"d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef\": container with ID starting with d3f57b108859d3e57fe3493e395446b713765134f359cd9a1733315c6016b3ef not found: ID does not exist"
Oct 30 20:47:34.073393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.073388    5002 scope.go:122] "RemoveContainer" containerID="bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047"
Oct 30 20:47:34.083754 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.083732    5002 scope.go:122] "RemoveContainer" containerID="b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c"
Oct 30 20:47:34.093559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.093525    5002 scope.go:122] "RemoveContainer" containerID="bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047"
Oct 30 20:47:34.093815 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:34.093794    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047\": container with ID starting with bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047 not found: ID does not exist" containerID="bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047"
Oct 30 20:47:34.093861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.093822    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047"} err="failed to get container status \"bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047\": rpc error: code = NotFound desc = could not find container \"bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047\": container with ID starting with bf5d94fc0c2607c82f586dad54d17a0f12f2af406fbff9238ace452a54a89047 not found: ID does not exist"
Oct 30 20:47:34.093861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.093839    5002 scope.go:122] "RemoveContainer" containerID="b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c"
Oct 30 20:47:34.094092 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:34.094077    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c\": container with ID starting with b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c not found: ID does not exist" containerID="b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c"
Oct 30 20:47:34.094134 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.094097    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c"} err="failed to get container status \"b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c\": rpc error: code = NotFound desc = could not find container \"b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c\": container with ID starting with b5e927fb174184d16518da1da9e5116f392746fdc1bdaf1b6809feab031ee11c not found: ID does not exist"
Oct 30 20:47:34.094134 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.094111    5002 scope.go:122] "RemoveContainer" containerID="4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e"
Oct 30 20:47:34.102969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.102954    5002 scope.go:122] "RemoveContainer" containerID="4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e"
Oct 30 20:47:34.103226 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:34.103207    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e\": container with ID starting with 4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e not found: ID does not exist" containerID="4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e"
Oct 30 20:47:34.103281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.103235    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e"} err="failed to get container status \"4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e\": rpc error: code = NotFound desc = could not find container \"4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e\": container with ID starting with 4bc50f05fba81de1b3d1310220da0b403428fed4a63e7490baf1c1ddcd9fc38e not found: ID does not exist"
Oct 30 20:47:34.203738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.203710    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"]
Oct 30 20:47:34.224186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.224159    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn"]
Oct 30 20:47:34.337839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.337811    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"]
Oct 30 20:47:34.341787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.341763    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd"]
Oct 30 20:47:34.452714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.452681    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"]
Oct 30 20:47:34.463615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.463593    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf"]
Oct 30 20:47:34.575039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.574958    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"]
Oct 30 20:47:34.578575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.578551    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh"]
Oct 30 20:47:34.688007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.687981    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"]
Oct 30 20:47:34.691978 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.691957    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs"]
Oct 30 20:47:34.923431 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:34.923328    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:35.739989 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:35.739950    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:35.931034 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:35.931002    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:35.968056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:35.968028    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" path="/var/lib/kubelet/pods/0e4eca8a-ec71-4ee5-b142-91786d5364e7/volumes"
Oct 30 20:47:35.968487 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:35.968472    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" path="/var/lib/kubelet/pods/1aacf197-1de8-403b-8e73-c3cfd73452a0/volumes"
Oct 30 20:47:35.968918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:35.968905    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" path="/var/lib/kubelet/pods/56204e2f-0989-4a1a-a7c2-dd9a707b827e/volumes"
Oct 30 20:47:35.969667 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:35.969652    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942ad519-417a-4b3f-be9c-5e104267f692" path="/var/lib/kubelet/pods/942ad519-417a-4b3f-be9c-5e104267f692/volumes"
Oct 30 20:47:35.970071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:35.970058    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" path="/var/lib/kubelet/pods/ce1f033c-7faa-4d52-8c3a-f8e50e5c0487/volumes"
Oct 30 20:47:36.379335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:36.379308    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:36.744929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:36.741992    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:37.116637 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:37.116552    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:47:37.404420 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:37.404342    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:47:37.404582 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:37.404342    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:47:37.760774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:37.760747    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:39.907342 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.907313    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"]
Oct 30 20:47:39.907973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.907959    5002 cpu_manager.go:414] "containerMap: removing container" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy"
Oct 30 20:47:39.908023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.907975    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy"
Oct 30 20:47:39.908023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908004    5002 cpu_manager.go:414] "containerMap: removing container" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy"
Oct 30 20:47:39.908023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908009    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy"
Oct 30 20:47:39.908023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908018    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy"
Oct 30 20:47:39.908023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908022    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy"
Oct 30 20:47:39.908154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908029    5002 cpu_manager.go:414] "containerMap: removing container" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy"
Oct 30 20:47:39.908154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908033    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy"
Oct 30 20:47:39.908154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908040    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy"
Oct 30 20:47:39.908154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908044    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy"
Oct 30 20:47:39.908154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908068    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy"
Oct 30 20:47:39.908154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908072    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy"
Oct 30 20:47:39.908352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908164    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy"
Oct 30 20:47:39.908352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908175    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e7cdb7-12ab-462a-9c63-b18e2038e659" containerName="istio-proxy"
Oct 30 20:47:39.908352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908179    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4eca8a-ec71-4ee5-b142-91786d5364e7" containerName="istio-proxy"
Oct 30 20:47:39.908352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908185    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="942ad519-417a-4b3f-be9c-5e104267f692" containerName="istio-proxy"
Oct 30 20:47:39.908352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908192    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy"
Oct 30 20:47:39.908352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908198    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aacf197-1de8-403b-8e73-c3cfd73452a0" containerName="istio-proxy"
Oct 30 20:47:39.908352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.908207    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1f033c-7faa-4d52-8c3a-f8e50e5c0487" containerName="istio-proxy"
Oct 30 20:47:39.913995 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.913974    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.965080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.965051    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.965205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.965133    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19f2f77-1092-4e4a-8a29-31b779197202-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.965205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.965191    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.965331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.965305    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.965391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.965375    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpfz\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-kube-api-access-jtpfz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.965449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.965414    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.965523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.965509    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:39.967020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.966998    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:47:39.967130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.967027    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:47:39.967130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.967028    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:47:39.967249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:39.967228    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jtvwq\""
Oct 30 20:47:40.024132 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.024104    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"]
Oct 30 20:47:40.066594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.066558    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.066748 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.066647    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19f2f77-1092-4e4a-8a29-31b779197202-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.066748 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.066707    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.066862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.066761    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.066862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.066811    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpfz\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-kube-api-access-jtpfz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.066994 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.066973    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.067122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.067108    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.067853 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.067833    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.069848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.069820    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.070677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.070656    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19f2f77-1092-4e4a-8a29-31b779197202-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.070802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.070785    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.071399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.071380    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.127601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.127580    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.127977 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.127955    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpfz\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-kube-api-access-jtpfz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zd4sq\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.238569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.229736    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:40.683645 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:40.683619    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19f2f77_1092_4e4a_8a29_31b779197202.slice/crio-e32826f05aa49bae74e5db046827043621d160caa547835104d74f90b39509c2 WatchSource:0}: Error finding container e32826f05aa49bae74e5db046827043621d160caa547835104d74f90b39509c2: Status 404 returned error can't find the container with id e32826f05aa49bae74e5db046827043621d160caa547835104d74f90b39509c2
Oct 30 20:47:40.683809 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:40.683785    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"]
Oct 30 20:47:41.089086 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:41.089049    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" event={"ID":"f19f2f77-1092-4e4a-8a29-31b779197202","Type":"ContainerStarted","Data":"e32826f05aa49bae74e5db046827043621d160caa547835104d74f90b39509c2"}
Oct 30 20:47:41.651972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:41.651941    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:42.314668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:42.314638    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:43.048130 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:43.048091    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:47:43.292990 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:43.292953    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:43.821389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:43.821327    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:44.691839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:44.691799    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:44.870841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:44.870810    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:44.887847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:44.887799    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:44.887965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:44.887889    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:45.132505 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:45.132462    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" event={"ID":"f19f2f77-1092-4e4a-8a29-31b779197202","Type":"ContainerStarted","Data":"6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf"}
Oct 30 20:47:45.132702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:45.132560    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:45.134146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:45.134116    5002 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:47:45.134323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:45.134179    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" podUID="f19f2f77-1092-4e4a-8a29-31b779197202" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:47:45.341168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:45.341123    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" podStartSLOduration=2.14054695 podStartE2EDuration="6.341110708s" totalImagesPullingTime="4.200563758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:45.341051089 +0000 UTC m=+1691.490102076" watchObservedRunningTime="2026-10-30 20:47:45.341110708 +0000 UTC m=+1691.490161691"
Oct 30 20:47:45.774834 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:45.774793    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:47:46.143738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:46.143649    5002 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:47:46.143738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:46.143705    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" podUID="f19f2f77-1092-4e4a-8a29-31b779197202" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:47:46.814687 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:46.811066    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:49.142946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:49.142919    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:47:49.153607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:49.153579    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:49.367185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:49.367149    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:49.556015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:49.555975    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:50.807197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:50.807162    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:51.585402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:51.585303    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:51.960379 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:51.960352    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:52.375375 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:52.375304    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:52.562994 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:52.562966    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:52.748954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:52.748894    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:52.960558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:52.951836    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"]
Oct 30 20:47:52.977605 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:52.962192    5002 cpu_manager.go:414] "containerMap: removing container" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy"
Oct 30 20:47:52.977874 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:52.977840    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56204e2f-0989-4a1a-a7c2-dd9a707b827e" containerName="istio-proxy"
Oct 30 20:47:52.989878 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:52.989850    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.064924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.064841    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-061bda-dockercfg-6bjqg\""
Oct 30 20:47:53.074579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.072819    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-061bda\""
Oct 30 20:47:53.103608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.101654    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd208ad-a560-4fe9-b756-a0bd171b70fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.103608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.101742    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nks\" (UniqueName: \"kubernetes.io/projected/4dd208ad-a560-4fe9-b756-a0bd171b70fb-kube-api-access-s7nks\") pod \"payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.108077 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.108046    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"]
Oct 30 20:47:53.173965 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:47:53.173936    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:47:53.207049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.207018    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd208ad-a560-4fe9-b756-a0bd171b70fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.207219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.207099    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nks\" (UniqueName: \"kubernetes.io/projected/4dd208ad-a560-4fe9-b756-a0bd171b70fb-kube-api-access-s7nks\") pod \"payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.207768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.207744    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd208ad-a560-4fe9-b756-a0bd171b70fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.265924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.265894    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nks\" (UniqueName: \"kubernetes.io/projected/4dd208ad-a560-4fe9-b756-a0bd171b70fb-kube-api-access-s7nks\") pod \"payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.417362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.417275    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:53.825283 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:47:53.825253    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd208ad_a560_4fe9_b756_a0bd171b70fb.slice/crio-35d7418d8b7fd732a357ece04a87baa2cf0b621cfb2e62c1b1f0446ce9752610 WatchSource:0}: Error finding container 35d7418d8b7fd732a357ece04a87baa2cf0b621cfb2e62c1b1f0446ce9752610: Status 404 returned error can't find the container with id 35d7418d8b7fd732a357ece04a87baa2cf0b621cfb2e62c1b1f0446ce9752610
Oct 30 20:47:53.825741 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:53.825719    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"]
Oct 30 20:47:54.210437 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:54.210402    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh" event={"ID":"4dd208ad-a560-4fe9-b756-a0bd171b70fb","Type":"ContainerStarted","Data":"5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03"}
Oct 30 20:47:54.210887 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:54.210458    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh" event={"ID":"4dd208ad-a560-4fe9-b756-a0bd171b70fb","Type":"ContainerStarted","Data":"35d7418d8b7fd732a357ece04a87baa2cf0b621cfb2e62c1b1f0446ce9752610"}
Oct 30 20:47:54.210887 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:54.210519    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:47:54.331469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:54.331402    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh" podStartSLOduration=2.331383786 podStartE2EDuration="2.331383786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:54.3299836 +0000 UTC m=+1700.479034587" watchObservedRunningTime="2026-10-30 20:47:54.331383786 +0000 UTC m=+1700.480434773"
Oct 30 20:47:56.169024 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:56.168995    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:56.389956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:56.389928    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:56.586691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:56.586660    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:57.545299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:57.545263    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:59.465262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:59.465221    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:59.662548 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:59.662507    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:59.997211 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:47:59.997173    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"]
Oct 30 20:48:00.020289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.020261    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.086400 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.086363    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"]
Oct 30 20:48:00.096787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.096757    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-034eba\""
Oct 30 20:48:00.097234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.097214    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-034eba-dockercfg-6crgz\""
Oct 30 20:48:00.118608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.118576    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff11802-8f1a-43ea-933c-96b271fc17eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.118762 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.118643    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfwv\" (UniqueName: \"kubernetes.io/projected/3ff11802-8f1a-43ea-933c-96b271fc17eb-kube-api-access-4sfwv\") pod \"payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.159055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.159029    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"]
Oct 30 20:48:00.164180 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.164157    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.219688 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.219660    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff11802-8f1a-43ea-933c-96b271fc17eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.220792 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.220767    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfwv\" (UniqueName: \"kubernetes.io/projected/3ff11802-8f1a-43ea-933c-96b271fc17eb-kube-api-access-4sfwv\") pod \"payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.221215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.220688    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff11802-8f1a-43ea-933c-96b271fc17eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.221932 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.221910    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"]
Oct 30 20:48:00.279146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.279087    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfwv\" (UniqueName: \"kubernetes.io/projected/3ff11802-8f1a-43ea-933c-96b271fc17eb-kube-api-access-4sfwv\") pod \"payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.321959 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.321930    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71c15ce-1584-4d9b-a8ae-f172e383b939-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.322146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.322119    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsx7\" (UniqueName: \"kubernetes.io/projected/c71c15ce-1584-4d9b-a8ae-f172e383b939-kube-api-access-zfsx7\") pod \"payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.387474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.387448    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:00.424168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.424136    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71c15ce-1584-4d9b-a8ae-f172e383b939-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.424302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.424235    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsx7\" (UniqueName: \"kubernetes.io/projected/c71c15ce-1584-4d9b-a8ae-f172e383b939-kube-api-access-zfsx7\") pod \"payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.424843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.424819    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71c15ce-1584-4d9b-a8ae-f172e383b939-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.488611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.488552    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsx7\" (UniqueName: \"kubernetes.io/projected/c71c15ce-1584-4d9b-a8ae-f172e383b939-kube-api-access-zfsx7\") pod \"payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.784626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.784591    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:00.794771 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:00.794751    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"]
Oct 30 20:48:00.799316 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:48:00.799288    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff11802_8f1a_43ea_933c_96b271fc17eb.slice/crio-877817192f3115c3376787c6a7a8ac9fa12bbeaa4c954f2035e2e3a2ae74b984 WatchSource:0}: Error finding container 877817192f3115c3376787c6a7a8ac9fa12bbeaa4c954f2035e2e3a2ae74b984: Status 404 returned error can't find the container with id 877817192f3115c3376787c6a7a8ac9fa12bbeaa4c954f2035e2e3a2ae74b984
Oct 30 20:48:00.847040 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:00.847004    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:01.182460 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.182433    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"]
Oct 30 20:48:01.182934 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:48:01.182911    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71c15ce_1584_4d9b_a8ae_f172e383b939.slice/crio-240e57614166a23c165c726cdbe81e22970fad2b0fb394720ba3cbbc84947ac3 WatchSource:0}: Error finding container 240e57614166a23c165c726cdbe81e22970fad2b0fb394720ba3cbbc84947ac3: Status 404 returned error can't find the container with id 240e57614166a23c165c726cdbe81e22970fad2b0fb394720ba3cbbc84947ac3
Oct 30 20:48:01.273328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.273298    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n" event={"ID":"c71c15ce-1584-4d9b-a8ae-f172e383b939","Type":"ContainerStarted","Data":"f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc"}
Oct 30 20:48:01.273496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.273479    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:01.273589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.273516    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n" event={"ID":"c71c15ce-1584-4d9b-a8ae-f172e383b939","Type":"ContainerStarted","Data":"240e57614166a23c165c726cdbe81e22970fad2b0fb394720ba3cbbc84947ac3"}
Oct 30 20:48:01.274867 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.274843    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4" event={"ID":"3ff11802-8f1a-43ea-933c-96b271fc17eb","Type":"ContainerStarted","Data":"1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9"}
Oct 30 20:48:01.274967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.274891    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4" event={"ID":"3ff11802-8f1a-43ea-933c-96b271fc17eb","Type":"ContainerStarted","Data":"877817192f3115c3376787c6a7a8ac9fa12bbeaa4c954f2035e2e3a2ae74b984"}
Oct 30 20:48:01.274967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.274936    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:01.393210 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.393163    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n" podStartSLOduration=1.393148939 podStartE2EDuration="1.393148939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:01.389319235 +0000 UTC m=+1707.538370224" watchObservedRunningTime="2026-10-30 20:48:01.393148939 +0000 UTC m=+1707.542199926"
Oct 30 20:48:01.512438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:01.512382    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4" podStartSLOduration=2.512364256 podStartE2EDuration="2.512364256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:01.509957589 +0000 UTC m=+1707.659008591" watchObservedRunningTime="2026-10-30 20:48:01.512364256 +0000 UTC m=+1707.661415234"
Oct 30 20:48:02.365302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:02.365260    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:02.553161 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:02.553131    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:02.745690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:02.745661    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:03.189365 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:03.189268    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:05.052113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:05.052085    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:48:05.224427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:05.224400    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:48:06.904822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:06.904791    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"]
Oct 30 20:48:06.909929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:06.909911    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:06.915953 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:06.915930    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vk9s\" (UniqueName: \"kubernetes.io/projected/f6883e6c-3852-4858-9724-8e523a3783d2-kube-api-access-7vk9s\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:06.916089 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:06.916039    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6883e6c-3852-4858-9724-8e523a3783d2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:06.916165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:06.916144    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:06.916285 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:06.916268    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:06.968167 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:06.968147    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"]
Oct 30 20:48:07.012606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.012576    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-99c030-serving-cert\""
Oct 30 20:48:07.016901 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.016873    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vk9s\" (UniqueName: \"kubernetes.io/projected/f6883e6c-3852-4858-9724-8e523a3783d2-kube-api-access-7vk9s\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.017113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.017082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6883e6c-3852-4858-9724-8e523a3783d2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.017243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.017222    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.017328 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:07.017311    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-serving-cert: secret "maas-api-e2e-shared-a-w2-99c030-serving-cert" not found
Oct 30 20:48:07.017383 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:07.017371    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls podName:f6883e6c-3852-4858-9724-8e523a3783d2 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:07.517354347 +0000 UTC m=+1713.666405332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls") pod "maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr" (UID: "f6883e6c-3852-4858-9724-8e523a3783d2") : secret "maas-api-e2e-shared-a-w2-99c030-serving-cert" not found
Oct 30 20:48:07.017431 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.017390    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.017744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.017727    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6883e6c-3852-4858-9724-8e523a3783d2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.019853 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.019835    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.075257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.075227    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vk9s\" (UniqueName: \"kubernetes.io/projected/f6883e6c-3852-4858-9724-8e523a3783d2-kube-api-access-7vk9s\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.116016 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:07.115982    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:48:07.143430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.143401    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:07.329310 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.329282    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:07.524272 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.524237    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.526996 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.526968    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:07.620513 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.620423    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:07.825962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:07.825917    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:08.208815 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:48:08.208786    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6883e6c_3852_4858_9724_8e523a3783d2.slice/crio-d98bb0d57215ec58fcb6a988920bcb64894002aac38cd815b3f26930af2feebf WatchSource:0}: Error finding container d98bb0d57215ec58fcb6a988920bcb64894002aac38cd815b3f26930af2feebf: Status 404 returned error can't find the container with id d98bb0d57215ec58fcb6a988920bcb64894002aac38cd815b3f26930af2feebf
Oct 30 20:48:08.209131 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:08.208951    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"]
Oct 30 20:48:08.333426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:08.333393    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr" event={"ID":"f6883e6c-3852-4858-9724-8e523a3783d2","Type":"ContainerStarted","Data":"d98bb0d57215ec58fcb6a988920bcb64894002aac38cd815b3f26930af2feebf"}
Oct 30 20:48:09.344047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:09.344008    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr" event={"ID":"f6883e6c-3852-4858-9724-8e523a3783d2","Type":"ContainerStarted","Data":"e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719"}
Oct 30 20:48:09.344433 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:09.344121    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:09.468012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:09.467962    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr" podStartSLOduration=2.57544388 podStartE2EDuration="3.467948912s" totalImagesPullingTime="892.505032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:09.464591283 +0000 UTC m=+1715.613642268" watchObservedRunningTime="2026-10-30 20:48:09.467948912 +0000 UTC m=+1715.616999898"
Oct 30 20:48:10.124940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:10.124909    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"]
Oct 30 20:48:10.125177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:10.125141    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy" containerID="cri-o://cbe6deb35ae963f380b419ae3b92156018fef5c9d9c8c692f0a9d8ce80660cbe" gracePeriod=30
Oct 30 20:48:10.174725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:10.174692    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"]
Oct 30 20:48:10.174907 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:10.174886    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy" containerID="cri-o://76194b177946df34f4fe6387f313300d46aa5239b4e16de80c68c86b3feb6fcb" gracePeriod=30
Oct 30 20:48:12.123981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:12.123924    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:12.284543 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:12.284504    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:48:12.284702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:12.284596    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:48:13.273816 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:13.273782    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:15.364312 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.359372    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:48:15.392249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.392089    5002 generic.go:356] "Generic (PLEG): container finished" podID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerID="cbe6deb35ae963f380b419ae3b92156018fef5c9d9c8c692f0a9d8ce80660cbe" exitCode=0
Oct 30 20:48:15.392249 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.392161    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" event={"ID":"4a838c0f-9a0d-4c8b-b937-0961fbb5b843","Type":"ContainerDied","Data":"cbe6deb35ae963f380b419ae3b92156018fef5c9d9c8c692f0a9d8ce80660cbe"}
Oct 30 20:48:15.394562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.394516    5002 generic.go:356] "Generic (PLEG): container finished" podID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerID="76194b177946df34f4fe6387f313300d46aa5239b4e16de80c68c86b3feb6fcb" exitCode=0
Oct 30 20:48:15.394830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.394590    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" event={"ID":"4a88b3c2-ef05-4597-a222-d46fcf731f9f","Type":"ContainerDied","Data":"76194b177946df34f4fe6387f313300d46aa5239b4e16de80c68c86b3feb6fcb"}
Oct 30 20:48:15.481775 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.481749    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:48:15.504079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.504054    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:48:15.530480 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530451    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-certs\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.530653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530509    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-credential-socket\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.530653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530557    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-token\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-token\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.530653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530611    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-socket\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.530653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530642    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-kube-api-access-76c2v\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-kube-api-access-76c2v\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.530872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530753    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istiod-ca-cert\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.530872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530778    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-certs" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.530872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530809    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-envoy\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.531013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530880    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-podinfo\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.531013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.530985    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-socket" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.531111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531029    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-data\") pod \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\" (UID: \"4a838c0f-9a0d-4c8b-b937-0961fbb5b843\") "
Oct 30 20:48:15.531184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531159    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-credential-socket" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.531258 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531236    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istiod-ca-cert" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:15.531632 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531607    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.531758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531633    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.531758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531649    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.531758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531663    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.531946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.531926    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-data" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.533977 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.533941    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-kube-api-access-76c2v" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "kube-api-access-76c2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.534431 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.534398    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-token" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.534598 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.534581    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-podinfo" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:48:15.535804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.535771    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-envoy" pod "4a838c0f-9a0d-4c8b-b937-0961fbb5b843" (UID: "4a838c0f-9a0d-4c8b-b937-0961fbb5b843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.635893    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-data\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.635940    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-socket\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636058    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-token\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-token\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636113    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-envoy\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636163    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-certs\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636227    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-podinfo\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636251    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-kube-api-access-6fbjr\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-kube-api-access-6fbjr\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636279    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istiod-ca-cert\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636310    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-credential-socket\") pod \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\" (UID: \"4a88b3c2-ef05-4597-a222-d46fcf731f9f\") "
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636818    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636832    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636841    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636850    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.637047 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.636860    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76c2v\" (UniqueName: \"kubernetes.io/projected/4a838c0f-9a0d-4c8b-b937-0961fbb5b843-kube-api-access-76c2v\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.637630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.637605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-credential-socket" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.637874 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.637854    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-data" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.639043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.639020    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istiod-ca-cert" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:15.639175 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.639077    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-certs" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.639175 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.639126    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-socket" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.641785 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.641759    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-podinfo" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:48:15.648685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.647645    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-envoy" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.648685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.647808    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-kube-api-access-6fbjr" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "kube-api-access-6fbjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.649200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.649162    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-token" pod "4a88b3c2-ef05-4597-a222-d46fcf731f9f" (UID: "4a88b3c2-ef05-4597-a222-d46fcf731f9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.738213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738174    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738213 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738213    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738227    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738241    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738254    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fbjr\" (UniqueName: \"kubernetes.io/projected/4a88b3c2-ef05-4597-a222-d46fcf731f9f-kube-api-access-6fbjr\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738292    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738306    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738319    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.738507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:15.738332    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a88b3c2-ef05-4597-a222-d46fcf731f9f-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.770084 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:15.770043    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:16.403270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.403196    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"
Oct 30 20:48:16.403270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.403210    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b" event={"ID":"4a838c0f-9a0d-4c8b-b937-0961fbb5b843","Type":"ContainerDied","Data":"33b106e0830df59b2a6c64217dfde115e91c2403dd9d7fb6412abff27ecc6c15"}
Oct 30 20:48:16.403825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.403282    5002 scope.go:122] "RemoveContainer" containerID="cbe6deb35ae963f380b419ae3b92156018fef5c9d9c8c692f0a9d8ce80660cbe"
Oct 30 20:48:16.404794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.404773    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6" event={"ID":"4a88b3c2-ef05-4597-a222-d46fcf731f9f","Type":"ContainerDied","Data":"4165ddf8e60592314a5fe4aa60735029a61272279aa0f637bb6397319c1c1127"}
Oct 30 20:48:16.404903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.404829    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"
Oct 30 20:48:16.416319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.416287    5002 scope.go:122] "RemoveContainer" containerID="76194b177946df34f4fe6387f313300d46aa5239b4e16de80c68c86b3feb6fcb"
Oct 30 20:48:16.582862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.582829    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"]
Oct 30 20:48:16.633676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.633650    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6"]
Oct 30 20:48:16.702301 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.702235    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"]
Oct 30 20:48:16.706422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.706400    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b"]
Oct 30 20:48:16.830909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:16.830866    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:17.969570 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:17.969517    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" path="/var/lib/kubelet/pods/4a838c0f-9a0d-4c8b-b937-0961fbb5b843/volumes"
Oct 30 20:48:17.970129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:17.970112    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" path="/var/lib/kubelet/pods/4a88b3c2-ef05-4597-a222-d46fcf731f9f/volumes"
Oct 30 20:48:21.350263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:21.350221    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:21.620204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:21.620124    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:21.834982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:21.834946    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:22.561328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:22.561288    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:22.885130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:22.885042    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:23.107829 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:23.107793    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:23.291471 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:23.291436    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:25.991262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:25.991229    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:27.354206 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:27.354169    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:29.268794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:29.268756    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:30.829457 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:30.829421    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:32.210701 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:32.210667    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:33.306362 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:33.306321    5002 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eca8a_ec71_4ee5_b142_91786d5364e7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f033c_7faa_4d52_8c3a_f8e50e5c0487.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:33.876563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:33.876502    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:37.002893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:37.002858    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:37.116857 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:37.116817    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:48:40.779557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:40.779496    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:41.697972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:41.697934    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:44.194744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.194643    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.221859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.221817    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 20:48:44.300700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.300657    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.467555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.464618    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.482337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.481610    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.629587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.627992    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:48:44.629587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.628325    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:48:44.759639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:44.759522    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:45.015925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.013935    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 20:48:45.088265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.088197    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 20:48:45.199555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.198684    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:45.199555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.199306    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:48:45.232560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.231618    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:45.232560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.232004    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:48:45.321614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.320609    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:45.321614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.320997    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:48:45.629612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.610129    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:45.629612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.610196    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:48:45.629612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.610774    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:48:45.629612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.610829    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:48:45.660568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.655625    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:45.660568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.656010    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:48:45.684774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.684312    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 20:48:45.710551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.702632    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:48:45.716607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.712347    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:48:45.736860 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.736412    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:48:45.771301 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.771277    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 20:48:45.805808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:45.805381    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 20:48:46.768718 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:46.768687    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:47.447933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:47.447895    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:50.554711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:50.554663    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:50.555256 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:50.554713    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:51.529732 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:51.529691    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:52.900210 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:52.900170    5002 generic.go:356] "Generic (PLEG): container finished" podID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerID="51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f" exitCode=0
Oct 30 20:48:52.901922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:52.900248    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" event={"ID":"d5a71733-90b6-46c8-8c95-80e0ed2d9251","Type":"ContainerDied","Data":"51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f"}
Oct 30 20:48:52.904350 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:52.904326    5002 scope.go:122] "RemoveContainer" containerID="51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f"
Oct 30 20:48:52.906704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:52.906613    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:52.906832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:52.906799    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:52.906894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:52.906854    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:53.932556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:53.928157    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" event={"ID":"d5a71733-90b6-46c8-8c95-80e0ed2d9251","Type":"ContainerStarted","Data":"f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd"}
Oct 30 20:48:54.259044 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:54.259006    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:48:54.259242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:54.259211    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.259288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:54.259268    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 20:48:55.259448 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.259413    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.259857 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.259479    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 20:48:55.968496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.968465    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"]
Oct 30 20:48:55.969151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.969133    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy"
Oct 30 20:48:55.969151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.969152    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy"
Oct 30 20:48:55.969268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.969166    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy"
Oct 30 20:48:55.969268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.969175    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy"
Oct 30 20:48:55.969329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.969301    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a838c0f-9a0d-4c8b-b937-0961fbb5b843" containerName="istio-proxy"
Oct 30 20:48:55.969329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.969310    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a88b3c2-ef05-4597-a222-d46fcf731f9f" containerName="istio-proxy"
Oct 30 20:48:55.973207 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:55.973190    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.045838 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.045805    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"]
Oct 30 20:48:56.077422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.077391    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-061bda-serving-cert\""
Oct 30 20:48:56.096568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.096504    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.096771 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.096749    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.096851 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.096786    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c2f71-0123-4dd2-8dee-7834e2cf7b32-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.097049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.097023    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxscb\" (UniqueName: \"kubernetes.io/projected/064c2f71-0123-4dd2-8dee-7834e2cf7b32-kube-api-access-fxscb\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.198290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.198261    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.198452 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.198298    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c2f71-0123-4dd2-8dee-7834e2cf7b32-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.198452 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.198374    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxscb\" (UniqueName: \"kubernetes.io/projected/064c2f71-0123-4dd2-8dee-7834e2cf7b32-kube-api-access-fxscb\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.198452 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.198400    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.198594 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:56.198576    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-serving-cert: secret "maas-api-e2e-shared-b-w1-061bda-serving-cert" not found
Oct 30 20:48:56.198677 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:48:56.198659    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls podName:064c2f71-0123-4dd2-8dee-7834e2cf7b32 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:56.698638645 +0000 UTC m=+1762.847689626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls") pod "maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn" (UID: "064c2f71-0123-4dd2-8dee-7834e2cf7b32") : secret "maas-api-e2e-shared-b-w1-061bda-serving-cert" not found
Oct 30 20:48:56.199130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.199113    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c2f71-0123-4dd2-8dee-7834e2cf7b32-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.200831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.200813    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.241496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.241075    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:56.262710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.260297    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.262710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.260354    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 20:48:56.263252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.263174    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxscb\" (UniqueName: \"kubernetes.io/projected/064c2f71-0123-4dd2-8dee-7834e2cf7b32-kube-api-access-fxscb\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.448115 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.447385    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:56.705876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.705795    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.708478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.708456    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:56.889052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:56.889018    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:57.259449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:57.259366    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.259449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:57.259431    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 20:48:57.340846 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:57.340807    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"]
Oct 30 20:48:57.977557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:57.977510    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn" event={"ID":"064c2f71-0123-4dd2-8dee-7834e2cf7b32","Type":"ContainerStarted","Data":"f67a9f302d24a0c284715a0fc51e1bc2ce6dc761d816d7a1ba62e102e2825650"}
Oct 30 20:48:58.259164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:58.259081    5002 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 20:48:58.259164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:58.259146    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 20:48:58.986441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:58.986402    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn" event={"ID":"064c2f71-0123-4dd2-8dee-7834e2cf7b32","Type":"ContainerStarted","Data":"c071454e33b2373f7c73f05c7261040a8f2066c98b375f4688c7a576a3e801a0"}
Oct 30 20:48:58.986855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:58.986466    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:48:59.106571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:59.106501    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn" podStartSLOduration=3.188920471 podStartE2EDuration="4.106487961s" totalImagesPullingTime="917.56749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:59.105573231 +0000 UTC m=+1765.254624217" watchObservedRunningTime="2026-10-30 20:48:59.106487961 +0000 UTC m=+1765.255538947"
Oct 30 20:48:59.263114 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:59.263044    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:48:59.263582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:59.263561    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:48:59.264250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:48:59.264235    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:49:01.203268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:01.203230    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:04.999824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:04.999791    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:49:05.671155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.671120    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"]
Oct 30 20:49:05.675877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.675855    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.751398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.751373    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.751596 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.751425    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311c46e-6d58-4217-96f1-f5d465fcc3a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.751677 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.751629    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsp6x\" (UniqueName: \"kubernetes.io/projected/e311c46e-6d58-4217-96f1-f5d465fcc3a4-kube-api-access-lsp6x\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.751738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.751680    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.780769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.780742    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-034eba-serving-cert\""
Oct 30 20:49:05.784921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.784899    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"]
Oct 30 20:49:05.853346 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.853318    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311c46e-6d58-4217-96f1-f5d465fcc3a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.853500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.853429    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsp6x\" (UniqueName: \"kubernetes.io/projected/e311c46e-6d58-4217-96f1-f5d465fcc3a4-kube-api-access-lsp6x\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.853500 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.853469    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.853635 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.853550    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.853700 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:49:05.853657    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-serving-cert: secret "maas-api-e2e-shared-a-w3-034eba-serving-cert" not found
Oct 30 20:49:05.853753 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:49:05.853727    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls podName:e311c46e-6d58-4217-96f1-f5d465fcc3a4 nodeName:}" failed. No retries permitted until 2026-10-30 20:49:06.35370882 +0000 UTC m=+1772.502759793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls") pod "maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7" (UID: "e311c46e-6d58-4217-96f1-f5d465fcc3a4") : secret "maas-api-e2e-shared-a-w3-034eba-serving-cert" not found
Oct 30 20:49:05.853933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.853911    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311c46e-6d58-4217-96f1-f5d465fcc3a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.856053 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.856033    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.912210 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.912185    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsp6x\" (UniqueName: \"kubernetes.io/projected/e311c46e-6d58-4217-96f1-f5d465fcc3a4-kube-api-access-lsp6x\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:05.912972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:05.912949    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:06.088009 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:06.087961    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:06.257983 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:06.257951    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:06.360615 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:06.360527    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:06.363088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:06.363057    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:06.590029 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:06.589999    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:06.979951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:06.979921    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"]
Oct 30 20:49:06.985421 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:49:06.985396    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode311c46e_6d58_4217_96f1_f5d465fcc3a4.slice/crio-0bbd0e1bb83c25401b3f32024b0ed883a894ef1daa35c7d1198c2581b7124bc8 WatchSource:0}: Error finding container 0bbd0e1bb83c25401b3f32024b0ed883a894ef1daa35c7d1198c2581b7124bc8: Status 404 returned error can't find the container with id 0bbd0e1bb83c25401b3f32024b0ed883a894ef1daa35c7d1198c2581b7124bc8
Oct 30 20:49:07.049710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:07.049677    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7" event={"ID":"e311c46e-6d58-4217-96f1-f5d465fcc3a4","Type":"ContainerStarted","Data":"0bbd0e1bb83c25401b3f32024b0ed883a894ef1daa35c7d1198c2581b7124bc8"}
Oct 30 20:49:07.117737 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:49:07.117697    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:49:08.059640 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:08.059607    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7" event={"ID":"e311c46e-6d58-4217-96f1-f5d465fcc3a4","Type":"ContainerStarted","Data":"01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2"}
Oct 30 20:49:08.059848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:08.059830    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:08.224237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:08.224193    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7" podStartSLOduration=2.291377014 podStartE2EDuration="3.224181624s" totalImagesPullingTime="932.80461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:08.220088333 +0000 UTC m=+1774.369139320" watchObservedRunningTime="2026-10-30 20:49:08.224181624 +0000 UTC m=+1774.373232610"
Oct 30 20:49:11.164869 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:11.164825    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:14.072871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:14.072843    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:49:14.691495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:14.691454    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:49:16.387333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:16.387284    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:18.705035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:18.704994    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:18.717631 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:18.717593    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:49:18.722639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:18.722608    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:18.867943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:18.867633    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:18.941448 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:18.941414    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"]
Oct 30 20:49:19.052667 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:19.052624    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.098653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:19.098613    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 20:49:19.407628 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:19.407064    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:19.609934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:19.609901    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:21.390721 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:21.390683    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:23.942641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:23.942600    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg" podUID="7372d973-3a84-4120-a0f2-a70f3355179c" containerName="payload-processing" containerID="cri-o://56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca" gracePeriod=25
Oct 30 20:49:24.222774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.222741    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:49:24.273509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.273479    5002 generic.go:356] "Generic (PLEG): container finished" podID="7372d973-3a84-4120-a0f2-a70f3355179c" containerID="56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca" exitCode=0
Oct 30 20:49:24.273964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.273582    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg" event={"ID":"7372d973-3a84-4120-a0f2-a70f3355179c","Type":"ContainerDied","Data":"56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca"}
Oct 30 20:49:24.273964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.273657    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg" event={"ID":"7372d973-3a84-4120-a0f2-a70f3355179c","Type":"ContainerDied","Data":"27a9223fd120b191779a264d09aa747aaa84fde8e76b43fa873aafe812edbcca"}
Oct 30 20:49:24.273964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.273689    5002 scope.go:122] "RemoveContainer" containerID="56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca"
Oct 30 20:49:24.273964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.273748    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"
Oct 30 20:49:24.284215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.284199    5002 scope.go:122] "RemoveContainer" containerID="56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca"
Oct 30 20:49:24.284485 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:49:24.284465    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca\": container with ID starting with 56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca not found: ID does not exist" containerID="56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca"
Oct 30 20:49:24.284567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.284497    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca"} err="failed to get container status \"56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca\": rpc error: code = NotFound desc = could not find container \"56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca\": container with ID starting with 56c1e66679a213e147b3d7739c53bbd6a5377a2ecb09116a3f447d43fbd872ca not found: ID does not exist"
Oct 30 20:49:24.298147 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.298125    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7372d973-3a84-4120-a0f2-a70f3355179c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372d973-3a84-4120-a0f2-a70f3355179c-plugins-config-volume\") pod \"7372d973-3a84-4120-a0f2-a70f3355179c\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") "
Oct 30 20:49:24.298363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.298349    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7372d973-3a84-4120-a0f2-a70f3355179c-kube-api-access-f278m\" (UniqueName: \"kubernetes.io/projected/7372d973-3a84-4120-a0f2-a70f3355179c-kube-api-access-f278m\") pod \"7372d973-3a84-4120-a0f2-a70f3355179c\" (UID: \"7372d973-3a84-4120-a0f2-a70f3355179c\") "
Oct 30 20:49:24.298496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.298475    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7372d973-3a84-4120-a0f2-a70f3355179c-plugins-config-volume" pod "7372d973-3a84-4120-a0f2-a70f3355179c" (UID: "7372d973-3a84-4120-a0f2-a70f3355179c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:24.298797 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.298777    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7372d973-3a84-4120-a0f2-a70f3355179c-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:24.300443 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.300421    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7372d973-3a84-4120-a0f2-a70f3355179c-kube-api-access-f278m" pod "7372d973-3a84-4120-a0f2-a70f3355179c" (UID: "7372d973-3a84-4120-a0f2-a70f3355179c"). InnerVolumeSpecName "kube-api-access-f278m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:24.399895 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.399866    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f278m\" (UniqueName: \"kubernetes.io/projected/7372d973-3a84-4120-a0f2-a70f3355179c-kube-api-access-f278m\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:24.699576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.699545    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"]
Oct 30 20:49:24.707079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:24.707054    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg"]
Oct 30 20:49:25.938586 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:25.931357    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:25.976505 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:25.976470    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7372d973-3a84-4120-a0f2-a70f3355179c" path="/var/lib/kubelet/pods/7372d973-3a84-4120-a0f2-a70f3355179c/volumes"
Oct 30 20:49:26.137039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:26.136994    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:26.791221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:26.791190    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:26.996153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:26.995977    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:30.271078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:30.271049    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:32.110597 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:32.110554    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:32.629014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:32.628983    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:34.223723 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:34.223682    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:34.253119 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:34.253097    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:49:34.260758 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:34.260731    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:49:34.405203 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:34.405171    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:34.756469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:34.756432    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:35.017432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.017349    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:35.144993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.144956    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"]
Oct 30 20:49:35.145731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.145709    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7372d973-3a84-4120-a0f2-a70f3355179c" containerName="payload-processing"
Oct 30 20:49:35.145731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.145731    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7372d973-3a84-4120-a0f2-a70f3355179c" containerName="payload-processing"
Oct 30 20:49:35.145915 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.145847    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7372d973-3a84-4120-a0f2-a70f3355179c" containerName="payload-processing"
Oct 30 20:49:35.151051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.151026    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.222197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.222169    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"]
Oct 30 20:49:35.260580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.260551    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-ab9c38\""
Oct 30 20:49:35.263629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.261317    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-ab9c38-dockercfg-r55lx\""
Oct 30 20:49:35.263824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.262473    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.264022 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.264004    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnrl\" (UniqueName: \"kubernetes.io/projected/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-kube-api-access-sbnrl\") pod \"payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.366419 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.365459    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.366678 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.366655    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnrl\" (UniqueName: \"kubernetes.io/projected/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-kube-api-access-sbnrl\") pod \"payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.367436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.367409    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.428001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.427972    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnrl\" (UniqueName: \"kubernetes.io/projected/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-kube-api-access-sbnrl\") pod \"payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.466967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.466941    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:35.848715 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:49:35.848683    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da68d8f_12b1_48dd_8e1c_81cdb0db1ab2.slice/crio-57d682cb6ec4c2b74dd325e61c6f2fec41cd8b44d1ad7fe3d857aeba360bf8a3 WatchSource:0}: Error finding container 57d682cb6ec4c2b74dd325e61c6f2fec41cd8b44d1ad7fe3d857aeba360bf8a3: Status 404 returned error can't find the container with id 57d682cb6ec4c2b74dd325e61c6f2fec41cd8b44d1ad7fe3d857aeba360bf8a3
Oct 30 20:49:35.848841 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:35.848721    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"]
Oct 30 20:49:36.256875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:36.256840    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:36.415594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:36.415556    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp" event={"ID":"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2","Type":"ContainerStarted","Data":"6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765"}
Oct 30 20:49:36.416012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:36.415627    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:36.416012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:36.415653    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp" event={"ID":"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2","Type":"ContainerStarted","Data":"57d682cb6ec4c2b74dd325e61c6f2fec41cd8b44d1ad7fe3d857aeba360bf8a3"}
Oct 30 20:49:36.460592 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:36.460518    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:36.587236 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:36.587143    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp" podStartSLOduration=1.587129112 podStartE2EDuration="1.587129112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:36.53541004 +0000 UTC m=+1802.684461038" watchObservedRunningTime="2026-10-30 20:49:36.587129112 +0000 UTC m=+1802.736180099"
Oct 30 20:49:36.682182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:36.682130    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:37.116890 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:49:37.116842    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:49:37.912653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:37.912615    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:39.709798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:39.709715    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:39.898330 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:39.898293    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:43.239563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:43.235228    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:47.426875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:47.426844    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:49:47.644135 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:47.644104    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:49.423831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:49.423796    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:49.642630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:49.642595    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:50.209332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:50.209300    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:53.332578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:53.332500    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:49:54.797018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:54.796960    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:56.969164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:56.969125    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:57.206470 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:57.206440    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:57.480504 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:57.480465    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:58.341983 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:58.341952    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:59.041198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:59.041165    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:59.288090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:49:59.288062    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:00.027661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:00.027629    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:03.550057 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:03.550019    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:03.776166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:03.776129    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:04.985902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:04.985858    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:07.121574 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:50:07.119153    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:50:07.473055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:07.473010    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:07.678026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:07.677990    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:10.480260 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:10.480206    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:15.580337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:15.580231    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:17.635331 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:17.635293    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:19.000871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:19.000837    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:19.682966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:19.682907    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:19.899985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:19.899931    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:20.113313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:20.113217    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:20.961024 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:20.960986    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:22.333391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:22.333345    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:23.906049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:23.906017    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:24.299048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:24.299015    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:25.704228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:25.704201    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:26.210228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:26.210190    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:29.070936 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:29.070904    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:29.437020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:29.436937    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:29.647923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:29.647889    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:29.942523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:29.942484    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:30.093640 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.093606    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"]
Oct 30 20:50:30.105859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.105838    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"]
Oct 30 20:50:30.106050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.105955    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.158731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.158701    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3f09ec-dockercfg-2fr9r\""
Oct 30 20:50:30.209241 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.209223    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3f09ec\""
Oct 30 20:50:30.219036 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.219012    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b628-a8be-417d-a102-b2a644603d16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.219150 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.219044    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxr2m\" (UniqueName: \"kubernetes.io/projected/fca7b628-a8be-417d-a102-b2a644603d16-kube-api-access-zxr2m\") pod \"payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.320308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.320281    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b628-a8be-417d-a102-b2a644603d16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.320441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.320316    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxr2m\" (UniqueName: \"kubernetes.io/projected/fca7b628-a8be-417d-a102-b2a644603d16-kube-api-access-zxr2m\") pod \"payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.321079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.321060    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b628-a8be-417d-a102-b2a644603d16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.378699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.378666    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxr2m\" (UniqueName: \"kubernetes.io/projected/fca7b628-a8be-417d-a102-b2a644603d16-kube-api-access-zxr2m\") pod \"payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.424773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.424747    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:30.433823 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.433800    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:30.816078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:30.816037    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"]
Oct 30 20:50:31.099607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:31.099453    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws" event={"ID":"fca7b628-a8be-417d-a102-b2a644603d16","Type":"ContainerStarted","Data":"79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7"}
Oct 30 20:50:31.099607 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:31.099521    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws" event={"ID":"fca7b628-a8be-417d-a102-b2a644603d16","Type":"ContainerStarted","Data":"4a8a72b62c67768a6e0257cf45838a5b8f0c643dba5f0480b76b6b33d174f52b"}
Oct 30 20:50:31.100070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:31.099617    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:31.235422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:31.235365    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws" podStartSLOduration=2.235350121 podStartE2EDuration="2.235350121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:31.230795075 +0000 UTC m=+1857.379846076" watchObservedRunningTime="2026-10-30 20:50:31.235350121 +0000 UTC m=+1857.384401502"
Oct 30 20:50:31.266297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:31.266261    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:31.579228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:31.579193    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:31.860828 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:31.860736    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:34.565053 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:34.565023    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:36.310741 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:36.310712    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:36.517511 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:36.517480    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:36.735812 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:36.735768    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:37.119658 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:50:37.119569    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:50:39.763920 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:39.763834    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:40.019993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:40.019906    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:42.023292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:42.023252    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:42.108927 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:42.108898    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:50:46.491385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.491343    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"]
Oct 30 20:50:46.497397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.497372    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.556286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.556258    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"]
Oct 30 20:50:46.600195 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.600168    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-ab9c38-serving-cert\""
Oct 30 20:50:46.628027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.627987    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.628352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.628334    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.628518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.628501    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.628689 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.628674    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979r2\" (UniqueName: \"kubernetes.io/projected/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-kube-api-access-979r2\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.733205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.730203    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.734474 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.734440    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.734641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.734560    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.734641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.734601    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979r2\" (UniqueName: \"kubernetes.io/projected/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-kube-api-access-979r2\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.736306 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.736278    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.738067 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:50:46.738047    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-serving-cert: secret "maas-api-e2e-shared-b-w5-ab9c38-serving-cert" not found
Oct 30 20:50:46.738250 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:50:46.738237    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls podName:b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd nodeName:}" failed. No retries permitted until 2026-10-30 20:50:47.238217309 +0000 UTC m=+1873.387268291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls") pod "maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" (UID: "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd") : secret "maas-api-e2e-shared-b-w5-ab9c38-serving-cert" not found
Oct 30 20:50:46.738989 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.738964    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:46.753027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.752923    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.795021 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.794994    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-979r2\" (UniqueName: \"kubernetes.io/projected/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-kube-api-access-979r2\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:46.946308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:46.946274    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:47.242005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:47.241977    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:47.244717 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:47.244688    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:47.411380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:47.411348    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:47.799290 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:50:47.799263    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98af6c7_b0ba_42f5_9a70_16d7a4ee03dd.slice/crio-32890b99dea8d245832045adc1b1657c8477388c468d59c9a18ca794ac731497 WatchSource:0}: Error finding container 32890b99dea8d245832045adc1b1657c8477388c468d59c9a18ca794ac731497: Status 404 returned error can't find the container with id 32890b99dea8d245832045adc1b1657c8477388c468d59c9a18ca794ac731497
Oct 30 20:50:47.800399 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:47.800370    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"]
Oct 30 20:50:48.229806 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:48.229774    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" event={"ID":"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd","Type":"ContainerStarted","Data":"32890b99dea8d245832045adc1b1657c8477388c468d59c9a18ca794ac731497"}
Oct 30 20:50:49.238919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:49.238883    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" event={"ID":"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd","Type":"ContainerStarted","Data":"bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123"}
Oct 30 20:50:49.239335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:49.239115    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:49.329307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:49.329277    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:49.361942 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:49.361894    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" podStartSLOduration=2.561136413 podStartE2EDuration="3.361882602s" totalImagesPullingTime="800.746189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:49.358782467 +0000 UTC m=+1875.507833458" watchObservedRunningTime="2026-10-30 20:50:49.361882602 +0000 UTC m=+1875.510933588"
Oct 30 20:50:51.453049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:51.453012    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:50:51.805431 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:51.805396    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:54.524557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:54.523676    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:54.524557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:54.523783    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:54.568558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:54.564824    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:54.568558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:54.564929    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:54.671763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:54.671723    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:54.752558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:54.749598    5002 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:54.774889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:54.774808    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:55.080250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:55.080058    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:55.253581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:55.253518    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:50:56.745183 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:56.745150    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:56.940322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:56.940285    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:57.129579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:50:57.129488    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:01.813793 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:01.813749    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:06.547266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:06.547231    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:07.116711 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:51:07.116676    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:51:11.399163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:11.399130    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:16.397696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:16.397665    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:21.768738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:21.768700    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:26.277477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.277442    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"]
Oct 30 20:51:26.281648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.281629    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.336179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.336142    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c3d13a-serving-cert\""
Oct 30 20:51:26.355988 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.355957    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbhgq\" (UniqueName: \"kubernetes.io/projected/8a024071-e692-4a4e-9fc8-65a4898c7c09-kube-api-access-lbhgq\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.356136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.356006    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-metrics-certs\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.356136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.356032    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.356263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.356218    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a024071-e692-4a4e-9fc8-65a4898c7c09-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.390491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.390465    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"]
Oct 30 20:51:26.457671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.457643    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhgq\" (UniqueName: \"kubernetes.io/projected/8a024071-e692-4a4e-9fc8-65a4898c7c09-kube-api-access-lbhgq\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.457826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.457680    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-metrics-certs\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.457826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.457707    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.457925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.457837    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a024071-e692-4a4e-9fc8-65a4898c7c09-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.457925 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:51:26.457879    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-serving-cert: secret "maas-api-e2e-api-a-c3d13a-serving-cert" not found
Oct 30 20:51:26.458031 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:51:26.457944    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls podName:8a024071-e692-4a4e-9fc8-65a4898c7c09 nodeName:}" failed. No retries permitted until 2026-10-30 20:51:26.957922733 +0000 UTC m=+1913.106973700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls") pod "maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2" (UID: "8a024071-e692-4a4e-9fc8-65a4898c7c09") : secret "maas-api-e2e-api-a-c3d13a-serving-cert" not found
Oct 30 20:51:26.458476 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.458451    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a024071-e692-4a4e-9fc8-65a4898c7c09-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.460158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.460140    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-metrics-certs\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.513166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.513138    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:26.515965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.515939    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbhgq\" (UniqueName: \"kubernetes.io/projected/8a024071-e692-4a4e-9fc8-65a4898c7c09-kube-api-access-lbhgq\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.713432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.713401    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:26.964588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.964513    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:26.967097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:26.967074    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls\") pod \"maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:27.195839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:27.195811    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:27.586664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:27.586633    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"]
Oct 30 20:51:27.587093 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:51:27.586875    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a024071_e692_4a4e_9fc8_65a4898c7c09.slice/crio-b09e3f6ef86d14f9b86bfab98b602b085eed7abbe47538e7570d35c69830661e WatchSource:0}: Error finding container b09e3f6ef86d14f9b86bfab98b602b085eed7abbe47538e7570d35c69830661e: Status 404 returned error can't find the container with id b09e3f6ef86d14f9b86bfab98b602b085eed7abbe47538e7570d35c69830661e
Oct 30 20:51:28.542571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:28.542524    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"]
Oct 30 20:51:28.596900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:28.596860    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2" event={"ID":"8a024071-e692-4a4e-9fc8-65a4898c7c09","Type":"ContainerStarted","Data":"1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345"}
Oct 30 20:51:28.597243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:28.596922    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2" event={"ID":"8a024071-e692-4a4e-9fc8-65a4898c7c09","Type":"ContainerStarted","Data":"b09e3f6ef86d14f9b86bfab98b602b085eed7abbe47538e7570d35c69830661e"}
Oct 30 20:51:28.597243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:28.597008    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:28.650793 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:28.650761    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"]
Oct 30 20:51:28.902381 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:28.902288    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2" podStartSLOduration=2.016773615 podStartE2EDuration="2.902274977s" totalImagesPullingTime="885.501362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:28.89818224 +0000 UTC m=+1915.047233227" watchObservedRunningTime="2026-10-30 20:51:28.902274977 +0000 UTC m=+1915.051325964"
Oct 30 20:51:28.966709 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:28.966676    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:29.163441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:29.163351    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:31.643030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:31.642970    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:33.542939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:33.542893    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" podUID="18c4bbd4-c505-4b24-8f91-205f82874461" containerName="payload-processing" containerID="cri-o://c13c0688ed3983be2b2d93d3415d2c4107ec37a4c58eac8bfbbe4d57999bbb91" gracePeriod=25
Oct 30 20:51:33.651304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:33.651268    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" podUID="fb912d4f-67f8-4d21-83d5-64193673e2c2" containerName="payload-pre-processing" containerID="cri-o://e8ac986545159b4295b31a0e6aaf566c9269d51243ec587e0389bdf0d88e6b9a" gracePeriod=25
Oct 30 20:51:34.609377 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:34.609346    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:51:35.022390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:35.022357    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:36.516304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:36.516261    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:37.116867 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:51:37.116833    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:51:40.190587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:40.190512    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" podUID="18c4bbd4-c505-4b24-8f91-205f82874461" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:51:40.190975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:40.190585    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" podUID="fb912d4f-67f8-4d21-83d5-64193673e2c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:51:41.397862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:41.397829    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:46.186287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:46.186210    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:50.190509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:50.190470    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" podUID="fb912d4f-67f8-4d21-83d5-64193673e2c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:51:50.190509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:50.190492    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" podUID="18c4bbd4-c505-4b24-8f91-205f82874461" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:51:50.871974 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:50.871934    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:51.063184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:51.063153    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:51.260581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:51.260549    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:52.851940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:52.851906    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:53.525432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:53.518963    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:53.776894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:53.776817    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:53.807684 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:53.807456    5002 generic.go:356] "Generic (PLEG): container finished" podID="18c4bbd4-c505-4b24-8f91-205f82874461" containerID="c13c0688ed3983be2b2d93d3415d2c4107ec37a4c58eac8bfbbe4d57999bbb91" exitCode=0
Oct 30 20:51:53.807684 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:53.807605    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" event={"ID":"18c4bbd4-c505-4b24-8f91-205f82874461","Type":"ContainerDied","Data":"c13c0688ed3983be2b2d93d3415d2c4107ec37a4c58eac8bfbbe4d57999bbb91"}
Oct 30 20:51:53.834467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:53.834232    5002 generic.go:356] "Generic (PLEG): container finished" podID="fb912d4f-67f8-4d21-83d5-64193673e2c2" containerID="e8ac986545159b4295b31a0e6aaf566c9269d51243ec587e0389bdf0d88e6b9a" exitCode=0
Oct 30 20:51:53.834467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:53.834403    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" event={"ID":"fb912d4f-67f8-4d21-83d5-64193673e2c2","Type":"ContainerDied","Data":"e8ac986545159b4295b31a0e6aaf566c9269d51243ec587e0389bdf0d88e6b9a"}
Oct 30 20:51:53.908038 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:53.905847    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:51:54.031744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.031659    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c4bbd4-c505-4b24-8f91-205f82874461-kube-api-access-7zxj9\" (UniqueName: \"kubernetes.io/projected/18c4bbd4-c505-4b24-8f91-205f82874461-kube-api-access-7zxj9\") pod \"18c4bbd4-c505-4b24-8f91-205f82874461\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") "
Oct 30 20:51:54.031744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.031716    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c4bbd4-c505-4b24-8f91-205f82874461-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c4bbd4-c505-4b24-8f91-205f82874461-plugins-config-volume\") pod \"18c4bbd4-c505-4b24-8f91-205f82874461\" (UID: \"18c4bbd4-c505-4b24-8f91-205f82874461\") "
Oct 30 20:51:54.039735 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.039708    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c4bbd4-c505-4b24-8f91-205f82874461-plugins-config-volume" pod "18c4bbd4-c505-4b24-8f91-205f82874461" (UID: "18c4bbd4-c505-4b24-8f91-205f82874461"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:54.041460 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.041425    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:51:54.042936 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.042915    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c4bbd4-c505-4b24-8f91-205f82874461-kube-api-access-7zxj9" pod "18c4bbd4-c505-4b24-8f91-205f82874461" (UID: "18c4bbd4-c505-4b24-8f91-205f82874461"). InnerVolumeSpecName "kube-api-access-7zxj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:54.133777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.133746    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb912d4f-67f8-4d21-83d5-64193673e2c2-kube-api-access-xv66k\" (UniqueName: \"kubernetes.io/projected/fb912d4f-67f8-4d21-83d5-64193673e2c2-kube-api-access-xv66k\") pod \"fb912d4f-67f8-4d21-83d5-64193673e2c2\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") "
Oct 30 20:51:54.133950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.133934    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb912d4f-67f8-4d21-83d5-64193673e2c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb912d4f-67f8-4d21-83d5-64193673e2c2-plugins-config-volume\") pod \"fb912d4f-67f8-4d21-83d5-64193673e2c2\" (UID: \"fb912d4f-67f8-4d21-83d5-64193673e2c2\") "
Oct 30 20:51:54.134319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.134293    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb912d4f-67f8-4d21-83d5-64193673e2c2-plugins-config-volume" pod "fb912d4f-67f8-4d21-83d5-64193673e2c2" (UID: "fb912d4f-67f8-4d21-83d5-64193673e2c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:54.134587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.134503    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb912d4f-67f8-4d21-83d5-64193673e2c2-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:54.134587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.134516    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxj9\" (UniqueName: \"kubernetes.io/projected/18c4bbd4-c505-4b24-8f91-205f82874461-kube-api-access-7zxj9\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:54.134587 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.134526    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c4bbd4-c505-4b24-8f91-205f82874461-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:54.136050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.136028    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb912d4f-67f8-4d21-83d5-64193673e2c2-kube-api-access-xv66k" pod "fb912d4f-67f8-4d21-83d5-64193673e2c2" (UID: "fb912d4f-67f8-4d21-83d5-64193673e2c2"). InnerVolumeSpecName "kube-api-access-xv66k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:54.236001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.235968    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv66k\" (UniqueName: \"kubernetes.io/projected/fb912d4f-67f8-4d21-83d5-64193673e2c2-kube-api-access-xv66k\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:54.845098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.844998    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"
Oct 30 20:51:54.845098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.844997    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh" event={"ID":"18c4bbd4-c505-4b24-8f91-205f82874461","Type":"ContainerDied","Data":"fa4809c4729d12ebbd7912ee787d4c306c7bea569b5f6b84b4ed148270d1027b"}
Oct 30 20:51:54.845326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.845154    5002 scope.go:122] "RemoveContainer" containerID="c13c0688ed3983be2b2d93d3415d2c4107ec37a4c58eac8bfbbe4d57999bbb91"
Oct 30 20:51:54.847868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.847849    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"
Oct 30 20:51:54.847997 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.847931    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb" event={"ID":"fb912d4f-67f8-4d21-83d5-64193673e2c2","Type":"ContainerDied","Data":"d3fb06a6a7f81151738836c9bd60c40d2732e174778c05d1ad9548baa2691f61"}
Oct 30 20:51:54.863283 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.863260    5002 scope.go:122] "RemoveContainer" containerID="e8ac986545159b4295b31a0e6aaf566c9269d51243ec587e0389bdf0d88e6b9a"
Oct 30 20:51:54.989467 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:54.989437    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:51:55.043588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:55.037478    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"]
Oct 30 20:51:55.043588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:55.042460    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh"]
Oct 30 20:51:55.156757 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:55.156676    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"]
Oct 30 20:51:55.162939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:55.162908    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb"]
Oct 30 20:51:56.004681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:56.004642    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c4bbd4-c505-4b24-8f91-205f82874461" path="/var/lib/kubelet/pods/18c4bbd4-c505-4b24-8f91-205f82874461/volumes"
Oct 30 20:51:56.005854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:56.005829    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb912d4f-67f8-4d21-83d5-64193673e2c2" path="/var/lib/kubelet/pods/fb912d4f-67f8-4d21-83d5-64193673e2c2/volumes"
Oct 30 20:51:56.032610 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:56.032350    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:57.781684 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:57.781649    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:57.994756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:57.994711    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:58.293280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:51:58.293252    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:01.296019 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:01.295987    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:02.305052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.305021    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:02.492391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.492358    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:02.669636 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.669556    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"]
Oct 30 20:52:02.670294 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.670277    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fb912d4f-67f8-4d21-83d5-64193673e2c2" containerName="payload-pre-processing"
Oct 30 20:52:02.670337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.670296    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb912d4f-67f8-4d21-83d5-64193673e2c2" containerName="payload-pre-processing"
Oct 30 20:52:02.670337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.670306    5002 cpu_manager.go:414] "containerMap: removing container" podUID="18c4bbd4-c505-4b24-8f91-205f82874461" containerName="payload-processing"
Oct 30 20:52:02.670337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.670311    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c4bbd4-c505-4b24-8f91-205f82874461" containerName="payload-processing"
Oct 30 20:52:02.670456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.670431    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb912d4f-67f8-4d21-83d5-64193673e2c2" containerName="payload-pre-processing"
Oct 30 20:52:02.670456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.670438    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c4bbd4-c505-4b24-8f91-205f82874461" containerName="payload-processing"
Oct 30 20:52:02.675854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.675834    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:02.730517 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.730489    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"]
Oct 30 20:52:02.760913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.760859    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5xb\" (UniqueName: \"kubernetes.io/projected/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-kube-api-access-bc5xb\") pod \"payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:02.761086 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.760985    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:02.777095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.777064    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3f0723\""
Oct 30 20:52:02.777482 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.777461    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3f0723-dockercfg-7vvvh\""
Oct 30 20:52:02.863078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.863039    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5xb\" (UniqueName: \"kubernetes.io/projected/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-kube-api-access-bc5xb\") pod \"payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:02.885829 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.885796    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:02.891868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.887011    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:02.894109 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.894078    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"]
Oct 30 20:52:02.903304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.903262    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"]
Oct 30 20:52:02.903469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.903450    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:02.924836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.924776    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5xb\" (UniqueName: \"kubernetes.io/projected/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-kube-api-access-bc5xb\") pod \"payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:02.989634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:02.989603    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:03.095648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.095609    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bgb\" (UniqueName: \"kubernetes.io/projected/46a5c650-1c2b-49da-b4f0-74ef63f02530-kube-api-access-t9bgb\") pod \"payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") " pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:03.096049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.096020    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a5c650-1c2b-49da-b4f0-74ef63f02530-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") " pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:03.197296 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.197230    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bgb\" (UniqueName: \"kubernetes.io/projected/46a5c650-1c2b-49da-b4f0-74ef63f02530-kube-api-access-t9bgb\") pod \"payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") " pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:03.197469 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.197449    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a5c650-1c2b-49da-b4f0-74ef63f02530-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") " pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:03.198154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.198128    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a5c650-1c2b-49da-b4f0-74ef63f02530-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") " pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:03.255397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.255377    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bgb\" (UniqueName: \"kubernetes.io/projected/46a5c650-1c2b-49da-b4f0-74ef63f02530-kube-api-access-t9bgb\") pod \"payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") " pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:03.366183 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.366144    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"]
Oct 30 20:52:03.366636 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:52:03.366427    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441e00a2_43ce_41ef_a1cb_0a5e32bd5c19.slice/crio-00aa34c73cf4a44ece73960c424869e4ae73bc9d7ce60457b214caa4ca2189ab WatchSource:0}: Error finding container 00aa34c73cf4a44ece73960c424869e4ae73bc9d7ce60457b214caa4ca2189ab: Status 404 returned error can't find the container with id 00aa34c73cf4a44ece73960c424869e4ae73bc9d7ce60457b214caa4ca2189ab
Oct 30 20:52:03.527210 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.527182    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:03.904937 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.904904    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"]
Oct 30 20:52:03.905145 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:52:03.905123    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a5c650_1c2b_49da_b4f0_74ef63f02530.slice/crio-6291d3977d436e979457db14ae0dd6d17fa0d8b2ddab98cb5d3cbeb8e5f5a54e WatchSource:0}: Error finding container 6291d3977d436e979457db14ae0dd6d17fa0d8b2ddab98cb5d3cbeb8e5f5a54e: Status 404 returned error can't find the container with id 6291d3977d436e979457db14ae0dd6d17fa0d8b2ddab98cb5d3cbeb8e5f5a54e
Oct 30 20:52:03.936441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.936334    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm" event={"ID":"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19","Type":"ContainerStarted","Data":"aa94307b74bc3a349afac548bf08a1b475fa8681516de38c1af45716f428e1b3"}
Oct 30 20:52:03.936441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.936395    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm" event={"ID":"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19","Type":"ContainerStarted","Data":"00aa34c73cf4a44ece73960c424869e4ae73bc9d7ce60457b214caa4ca2189ab"}
Oct 30 20:52:03.936683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.936465    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:03.938076 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:03.938032    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h" event={"ID":"46a5c650-1c2b-49da-b4f0-74ef63f02530","Type":"ContainerStarted","Data":"6291d3977d436e979457db14ae0dd6d17fa0d8b2ddab98cb5d3cbeb8e5f5a54e"}
Oct 30 20:52:04.055365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:04.055317    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm" podStartSLOduration=2.055299856 podStartE2EDuration="2.055299856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:04.054253002 +0000 UTC m=+1950.203303988" watchObservedRunningTime="2026-10-30 20:52:04.055299856 +0000 UTC m=+1950.204350842"
Oct 30 20:52:04.947156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:04.947122    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h" event={"ID":"46a5c650-1c2b-49da-b4f0-74ef63f02530","Type":"ContainerStarted","Data":"b379cdb9b66412cfca094a4f5f72b438c4e59d26de06d4b961b0db577238e534"}
Oct 30 20:52:04.947665 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:04.947214    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:05.068878 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:05.068823    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h" podStartSLOduration=3.06880934 podStartE2EDuration="3.06880934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:05.066002917 +0000 UTC m=+1951.215053905" watchObservedRunningTime="2026-10-30 20:52:05.06880934 +0000 UTC m=+1951.217860326"
Oct 30 20:52:06.041877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:06.041846    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:06.227198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:06.227170    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:06.401867 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:06.401792    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:07.137087 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:07.137044    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:52:11.027843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:11.027801    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:12.494480 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:12.494451    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:12.725368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:12.725340    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:14.950642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:14.950613    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:52:15.967971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:15.967941    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:52:16.000239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:16.000205    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:19.314376 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:19.314340    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:20.812524 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:20.812478    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:25.682440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:25.682402    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:27.101786 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:27.101713    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"]
Oct 30 20:52:27.102217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:27.101899    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy" containerID="cri-o://e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543" gracePeriod=30
Oct 30 20:52:30.395304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:30.395266    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"]
Oct 30 20:52:30.396159 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:30.395515    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy" containerID="cri-o://24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a" gracePeriod=30
Oct 30 20:52:30.977960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:30.977917    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:32.395629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.395596    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:52:32.486724 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.486694    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-podinfo\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.486871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.486775    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-envoy\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.486917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.486882    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-token\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-token\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.487003 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.486987    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-kube-api-access-fq8w6\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-kube-api-access-fq8w6\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.487070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487047    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c790f12-d632-4162-85f1-8222c41b4d3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c790f12-d632-4162-85f1-8222c41b4d3f-istiod-ca-cert\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.487134 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487076    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-credential-socket\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.487190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487176    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-socket\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.487275 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487249    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-data\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.487334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487310    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-certs\") pod \"3c790f12-d632-4162-85f1-8222c41b4d3f\" (UID: \"3c790f12-d632-4162-85f1-8222c41b4d3f\") "
Oct 30 20:52:32.487682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487612    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c790f12-d632-4162-85f1-8222c41b4d3f-istiod-ca-cert" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:32.487828 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487797    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-data" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.487910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.487871    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-socket" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.488043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.488020    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-certs" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.488119 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.488101    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-credential-socket" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.488343 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.488321    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.488417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.488348    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.488417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.488363    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.488417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.488376    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c790f12-d632-4162-85f1-8222c41b4d3f-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.488417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.488390    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.489191 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.489166    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-podinfo" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:32.489306 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.489229    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-envoy" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.489306 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.489281    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-token" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.489429 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.489323    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-kube-api-access-fq8w6" pod "3c790f12-d632-4162-85f1-8222c41b4d3f" (UID: "3c790f12-d632-4162-85f1-8222c41b4d3f"). InnerVolumeSpecName "kube-api-access-fq8w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.590191 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.590158    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.590329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.590196    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.590329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.590210    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.590329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:32.590226    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8w6\" (UniqueName: \"kubernetes.io/projected/3c790f12-d632-4162-85f1-8222c41b4d3f-kube-api-access-fq8w6\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:33.176667 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.176635    5002 generic.go:356] "Generic (PLEG): container finished" podID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerID="e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543" exitCode=0
Oct 30 20:52:33.176893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.176698    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"
Oct 30 20:52:33.176893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.176728    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" event={"ID":"3c790f12-d632-4162-85f1-8222c41b4d3f","Type":"ContainerDied","Data":"e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543"}
Oct 30 20:52:33.176893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.176784    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl" event={"ID":"3c790f12-d632-4162-85f1-8222c41b4d3f","Type":"ContainerDied","Data":"61a47037cbbfe469223e4fa156881838c7c8fc6549b886a8a5d3b78e9e385b8e"}
Oct 30 20:52:33.176893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.176811    5002 scope.go:122] "RemoveContainer" containerID="e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543"
Oct 30 20:52:33.188297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.188282    5002 scope.go:122] "RemoveContainer" containerID="e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543"
Oct 30 20:52:33.188582 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:33.188553    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543\": container with ID starting with e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543 not found: ID does not exist" containerID="e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543"
Oct 30 20:52:33.188693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.188590    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543"} err="failed to get container status \"e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543\": rpc error: code = NotFound desc = could not find container \"e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543\": container with ID starting with e7f540bad182171208b84ead23fee6eb1fe3d6b625f82cca91a7ceb3982a6543 not found: ID does not exist"
Oct 30 20:52:33.396018 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.395993    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"]
Oct 30 20:52:33.448709 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.448651    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl"]
Oct 30 20:52:33.975330 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:33.975300    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" path="/var/lib/kubelet/pods/3c790f12-d632-4162-85f1-8222c41b4d3f/volumes"
Oct 30 20:52:35.239050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.238991    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:35.425876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.425846    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:35.707211 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.707186    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:52:35.831158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831096    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-credential-socket\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831131    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-podinfo\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831154    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-envoy\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831189    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-data\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831211    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-certs\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831243    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-token\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-token\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831316    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-socket\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831383    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-credential-socket" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831433    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-kube-api-access-7x8cw\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-kube-api-access-7x8cw\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831484    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9823d53-c5da-406a-a02c-1509bf0d2edc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9823d53-c5da-406a-a02c-1509bf0d2edc-istiod-ca-cert\") pod \"f9823d53-c5da-406a-a02c-1509bf0d2edc\" (UID: \"f9823d53-c5da-406a-a02c-1509bf0d2edc\") "
Oct 30 20:52:35.831646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.831606    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-socket" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.832063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.832001    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9823d53-c5da-406a-a02c-1509bf0d2edc-istiod-ca-cert" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.832127 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.832106    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.832181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.832134    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.832181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.832149    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9823d53-c5da-406a-a02c-1509bf0d2edc-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.832282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.832233    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-certs" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.832339 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.832292    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-data" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.833796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.833773    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-token" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.834277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.834246    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-kube-api-access-7x8cw" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "kube-api-access-7x8cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.834375 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.834353    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-envoy" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.834444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.834426    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-podinfo" pod "f9823d53-c5da-406a-a02c-1509bf0d2edc" (UID: "f9823d53-c5da-406a-a02c-1509bf0d2edc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:35.934965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.934928    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.934965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.934966    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x8cw\" (UniqueName: \"kubernetes.io/projected/f9823d53-c5da-406a-a02c-1509bf0d2edc-kube-api-access-7x8cw\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.935197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.934981    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.935197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.934990    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.935197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.934999    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.935197 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:35.935006    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9823d53-c5da-406a-a02c-1509bf0d2edc-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.204768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.204684    5002 generic.go:356] "Generic (PLEG): container finished" podID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerID="24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a" exitCode=0
Oct 30 20:52:36.204768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.204760    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"
Oct 30 20:52:36.205017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.204772    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" event={"ID":"f9823d53-c5da-406a-a02c-1509bf0d2edc","Type":"ContainerDied","Data":"24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a"}
Oct 30 20:52:36.205017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.204838    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv" event={"ID":"f9823d53-c5da-406a-a02c-1509bf0d2edc","Type":"ContainerDied","Data":"4d0940bdbf3e972b2dc719c2337d6cf15d1c2d4857371feb18aa384cd71f3e03"}
Oct 30 20:52:36.205017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.204867    5002 scope.go:122] "RemoveContainer" containerID="24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a"
Oct 30 20:52:36.219583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.219559    5002 scope.go:122] "RemoveContainer" containerID="24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a"
Oct 30 20:52:36.221004 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:36.220972    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a\": container with ID starting with 24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a not found: ID does not exist" containerID="24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a"
Oct 30 20:52:36.221152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.221012    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a"} err="failed to get container status \"24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a\": rpc error: code = NotFound desc = could not find container \"24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a\": container with ID starting with 24603a4787b92aeffdf7718afd13ee3626dbd4995846938ebe1fe76cb56fbb4a not found: ID does not exist"
Oct 30 20:52:36.226657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.226611    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:36.374083 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.374054    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"]
Oct 30 20:52:36.378432 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:36.378406    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv"]
Oct 30 20:52:37.117551 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:37.117496    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:52:37.969185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:37.969153    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" path="/var/lib/kubelet/pods/f9823d53-c5da-406a-a02c-1509bf0d2edc/volumes"
Oct 30 20:52:38.234447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:38.234374    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:38.431145 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:38.431058    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:41.394314 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:41.394259    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:43.208624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:43.208589    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"]
Oct 30 20:52:43.321731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:43.321693    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"]
Oct 30 20:52:43.660016 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:43.659979    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:43.837344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:43.837305    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:52:43.853372 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:43.853341    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:46.483976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:46.483942    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:47.038356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:47.038326    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"]
Oct 30 20:52:47.157419 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:47.157378    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"]
Oct 30 20:52:47.460549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:47.460509    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:47.645690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:47.645658    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:48.209294 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.209247    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66" podUID="62aad480-43f9-45dd-8f51-2924f7091faf" containerName="payload-processing" containerID="cri-o://350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a" gracePeriod=25
Oct 30 20:52:48.322307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.322275    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp" podUID="1e3ed774-c246-488d-828b-0bb4126a9fb0" containerName="payload-pre-processing" containerID="cri-o://b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5" gracePeriod=25
Oct 30 20:52:48.518178 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.512766    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:52:48.534427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.533447    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62aad480-43f9-45dd-8f51-2924f7091faf-kube-api-access-mtlpp\" (UniqueName: \"kubernetes.io/projected/62aad480-43f9-45dd-8f51-2924f7091faf-kube-api-access-mtlpp\") pod \"62aad480-43f9-45dd-8f51-2924f7091faf\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") "
Oct 30 20:52:48.534427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.533850    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62aad480-43f9-45dd-8f51-2924f7091faf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aad480-43f9-45dd-8f51-2924f7091faf-plugins-config-volume\") pod \"62aad480-43f9-45dd-8f51-2924f7091faf\" (UID: \"62aad480-43f9-45dd-8f51-2924f7091faf\") "
Oct 30 20:52:48.536273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.536241    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62aad480-43f9-45dd-8f51-2924f7091faf-plugins-config-volume" pod "62aad480-43f9-45dd-8f51-2924f7091faf" (UID: "62aad480-43f9-45dd-8f51-2924f7091faf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:48.632556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.628391    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62aad480-43f9-45dd-8f51-2924f7091faf-kube-api-access-mtlpp" pod "62aad480-43f9-45dd-8f51-2924f7091faf" (UID: "62aad480-43f9-45dd-8f51-2924f7091faf"). InnerVolumeSpecName "kube-api-access-mtlpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:48.638618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.637367    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aad480-43f9-45dd-8f51-2924f7091faf-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:48.638618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.637401    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlpp\" (UniqueName: \"kubernetes.io/projected/62aad480-43f9-45dd-8f51-2924f7091faf-kube-api-access-mtlpp\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:48.671550 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.671510    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:52:48.737923 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.737856    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3ed774-c246-488d-828b-0bb4126a9fb0-kube-api-access-gnt5v\" (UniqueName: \"kubernetes.io/projected/1e3ed774-c246-488d-828b-0bb4126a9fb0-kube-api-access-gnt5v\") pod \"1e3ed774-c246-488d-828b-0bb4126a9fb0\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") "
Oct 30 20:52:48.738090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.737997    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3ed774-c246-488d-828b-0bb4126a9fb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3ed774-c246-488d-828b-0bb4126a9fb0-plugins-config-volume\") pod \"1e3ed774-c246-488d-828b-0bb4126a9fb0\" (UID: \"1e3ed774-c246-488d-828b-0bb4126a9fb0\") "
Oct 30 20:52:48.738407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.738380    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3ed774-c246-488d-828b-0bb4126a9fb0-plugins-config-volume" pod "1e3ed774-c246-488d-828b-0bb4126a9fb0" (UID: "1e3ed774-c246-488d-828b-0bb4126a9fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:48.743484 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.743456    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3ed774-c246-488d-828b-0bb4126a9fb0-kube-api-access-gnt5v" pod "1e3ed774-c246-488d-828b-0bb4126a9fb0" (UID: "1e3ed774-c246-488d-828b-0bb4126a9fb0"). InnerVolumeSpecName "kube-api-access-gnt5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:48.839571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.839514    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3ed774-c246-488d-828b-0bb4126a9fb0-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:48.839571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:48.839571    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnt5v\" (UniqueName: \"kubernetes.io/projected/1e3ed774-c246-488d-828b-0bb4126a9fb0-kube-api-access-gnt5v\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:49.317143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.317104    5002 generic.go:356] "Generic (PLEG): container finished" podID="62aad480-43f9-45dd-8f51-2924f7091faf" containerID="350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a" exitCode=0
Oct 30 20:52:49.317322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.317145    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"
Oct 30 20:52:49.317322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.317188    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66" event={"ID":"62aad480-43f9-45dd-8f51-2924f7091faf","Type":"ContainerDied","Data":"350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a"}
Oct 30 20:52:49.317322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.317243    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66" event={"ID":"62aad480-43f9-45dd-8f51-2924f7091faf","Type":"ContainerDied","Data":"634c0eb0aca35b559ab3e70a4c731ddd8b4a9cb824cd31fec0a055f7fb2e7744"}
Oct 30 20:52:49.317322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.317269    5002 scope.go:122] "RemoveContainer" containerID="350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a"
Oct 30 20:52:49.318750 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.318729    5002 generic.go:356] "Generic (PLEG): container finished" podID="1e3ed774-c246-488d-828b-0bb4126a9fb0" containerID="b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5" exitCode=0
Oct 30 20:52:49.318832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.318786    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"
Oct 30 20:52:49.318832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.318810    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp" event={"ID":"1e3ed774-c246-488d-828b-0bb4126a9fb0","Type":"ContainerDied","Data":"b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5"}
Oct 30 20:52:49.318949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.318854    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp" event={"ID":"1e3ed774-c246-488d-828b-0bb4126a9fb0","Type":"ContainerDied","Data":"67f79b2290e01eb2f4a8ef2b2307b1d7650440cb991c966689561087c20b94bc"}
Oct 30 20:52:49.329660 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.329645    5002 scope.go:122] "RemoveContainer" containerID="350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a"
Oct 30 20:52:49.329906 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:49.329891    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a\": container with ID starting with 350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a not found: ID does not exist" containerID="350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a"
Oct 30 20:52:49.329954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.329913    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a"} err="failed to get container status \"350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a\": rpc error: code = NotFound desc = could not find container \"350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a\": container with ID starting with 350ae94fb0928bb51cb1e132cddbab12ac9531940310a5795575665eea38249a not found: ID does not exist"
Oct 30 20:52:49.329954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.329928    5002 scope.go:122] "RemoveContainer" containerID="b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5"
Oct 30 20:52:49.339977 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.339962    5002 scope.go:122] "RemoveContainer" containerID="b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5"
Oct 30 20:52:49.340221 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:49.340205    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5\": container with ID starting with b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5 not found: ID does not exist" containerID="b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5"
Oct 30 20:52:49.340269 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.340227    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5"} err="failed to get container status \"b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5\": rpc error: code = NotFound desc = could not find container \"b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5\": container with ID starting with b2b9e3e828cef5db8428eb0613dc5dd06e5c9feaee32eb64d8a6f4c4358839b5 not found: ID does not exist"
Oct 30 20:52:49.492578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.492489    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"]
Oct 30 20:52:49.541868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.541844    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66"]
Oct 30 20:52:49.616510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.616485    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"]
Oct 30 20:52:49.624784 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.624041    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp"]
Oct 30 20:52:49.967440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.967411    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3ed774-c246-488d-828b-0bb4126a9fb0" path="/var/lib/kubelet/pods/1e3ed774-c246-488d-828b-0bb4126a9fb0/volumes"
Oct 30 20:52:49.967809 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:49.967800    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62aad480-43f9-45dd-8f51-2924f7091faf" path="/var/lib/kubelet/pods/62aad480-43f9-45dd-8f51-2924f7091faf/volumes"
Oct 30 20:52:51.157303 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:51.157266    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:52.039437 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.039400    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl" podUID="5b1487eb-98e0-4f47-bd93-95ab47d1f626" containerName="payload-processing" containerID="cri-o://34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85" gracePeriod=25
Oct 30 20:52:52.158358 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.158330    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4" podUID="cd640b3a-cea1-499f-a2fb-5e118f46407d" containerName="payload-pre-processing" containerID="cri-o://7f4638d469dcfabe8a1fd0635d8aca55a59185a16e97396d2dda3353225b555b" gracePeriod=25
Oct 30 20:52:52.323274 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.323246    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:52:52.355356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.355197    5002 generic.go:356] "Generic (PLEG): container finished" podID="cd640b3a-cea1-499f-a2fb-5e118f46407d" containerID="7f4638d469dcfabe8a1fd0635d8aca55a59185a16e97396d2dda3353225b555b" exitCode=0
Oct 30 20:52:52.355503 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.355291    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4" event={"ID":"cd640b3a-cea1-499f-a2fb-5e118f46407d","Type":"ContainerDied","Data":"7f4638d469dcfabe8a1fd0635d8aca55a59185a16e97396d2dda3353225b555b"}
Oct 30 20:52:52.357151 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.357128    5002 generic.go:356] "Generic (PLEG): container finished" podID="5b1487eb-98e0-4f47-bd93-95ab47d1f626" containerID="34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85" exitCode=0
Oct 30 20:52:52.357278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.357197    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl" event={"ID":"5b1487eb-98e0-4f47-bd93-95ab47d1f626","Type":"ContainerDied","Data":"34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85"}
Oct 30 20:52:52.357278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.357249    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl" event={"ID":"5b1487eb-98e0-4f47-bd93-95ab47d1f626","Type":"ContainerDied","Data":"e40a9cc9d368eb9a49f6caeaf933d3bace620750e135c2672ee2576d7b6ffd41"}
Oct 30 20:52:52.357368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.357273    5002 scope.go:122] "RemoveContainer" containerID="34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85"
Oct 30 20:52:52.357368 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.357290    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"
Oct 30 20:52:52.369956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.369903    5002 scope.go:122] "RemoveContainer" containerID="34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85"
Oct 30 20:52:52.370336 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:52.370307    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85\": container with ID starting with 34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85 not found: ID does not exist" containerID="34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85"
Oct 30 20:52:52.370455 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.370348    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85"} err="failed to get container status \"34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85\": rpc error: code = NotFound desc = could not find container \"34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85\": container with ID starting with 34b62f89c9d950dbcc37f541d1202eca975b572e6493072f22806a174fa94c85 not found: ID does not exist"
Oct 30 20:52:52.380061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.380035    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1487eb-98e0-4f47-bd93-95ab47d1f626-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1487eb-98e0-4f47-bd93-95ab47d1f626-plugins-config-volume\") pod \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") "
Oct 30 20:52:52.380171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.380094    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1487eb-98e0-4f47-bd93-95ab47d1f626-kube-api-access-8wr2z\" (UniqueName: \"kubernetes.io/projected/5b1487eb-98e0-4f47-bd93-95ab47d1f626-kube-api-access-8wr2z\") pod \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\" (UID: \"5b1487eb-98e0-4f47-bd93-95ab47d1f626\") "
Oct 30 20:52:52.380440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.380403    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1487eb-98e0-4f47-bd93-95ab47d1f626-plugins-config-volume" pod "5b1487eb-98e0-4f47-bd93-95ab47d1f626" (UID: "5b1487eb-98e0-4f47-bd93-95ab47d1f626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:52.381088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.381063    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1487eb-98e0-4f47-bd93-95ab47d1f626-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:52.383618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.383581    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1487eb-98e0-4f47-bd93-95ab47d1f626-kube-api-access-8wr2z" pod "5b1487eb-98e0-4f47-bd93-95ab47d1f626" (UID: "5b1487eb-98e0-4f47-bd93-95ab47d1f626"). InnerVolumeSpecName "kube-api-access-8wr2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:52.431104 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.431076    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:52:52.481957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.481926    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd640b3a-cea1-499f-a2fb-5e118f46407d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd640b3a-cea1-499f-a2fb-5e118f46407d-plugins-config-volume\") pod \"cd640b3a-cea1-499f-a2fb-5e118f46407d\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") "
Oct 30 20:52:52.482426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.482136    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd640b3a-cea1-499f-a2fb-5e118f46407d-kube-api-access-tr8lx\" (UniqueName: \"kubernetes.io/projected/cd640b3a-cea1-499f-a2fb-5e118f46407d-kube-api-access-tr8lx\") pod \"cd640b3a-cea1-499f-a2fb-5e118f46407d\" (UID: \"cd640b3a-cea1-499f-a2fb-5e118f46407d\") "
Oct 30 20:52:52.482780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.482753    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wr2z\" (UniqueName: \"kubernetes.io/projected/5b1487eb-98e0-4f47-bd93-95ab47d1f626-kube-api-access-8wr2z\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:52.482894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.482839    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd640b3a-cea1-499f-a2fb-5e118f46407d-plugins-config-volume" pod "cd640b3a-cea1-499f-a2fb-5e118f46407d" (UID: "cd640b3a-cea1-499f-a2fb-5e118f46407d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:52.484865 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.484839    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd640b3a-cea1-499f-a2fb-5e118f46407d-kube-api-access-tr8lx" pod "cd640b3a-cea1-499f-a2fb-5e118f46407d" (UID: "cd640b3a-cea1-499f-a2fb-5e118f46407d"). InnerVolumeSpecName "kube-api-access-tr8lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:52.583644 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.583585    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd640b3a-cea1-499f-a2fb-5e118f46407d-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:52.583644 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.583610    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr8lx\" (UniqueName: \"kubernetes.io/projected/cd640b3a-cea1-499f-a2fb-5e118f46407d-kube-api-access-tr8lx\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:52.784483 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.784451    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"]
Oct 30 20:52:52.788036 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:52.788002    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl"]
Oct 30 20:52:53.367095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:53.367013    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"
Oct 30 20:52:53.367095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:53.367049    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4" event={"ID":"cd640b3a-cea1-499f-a2fb-5e118f46407d","Type":"ContainerDied","Data":"ba76b4e52eeb592bf187e73f23c712a972ebfcf361727c34e67ab4e69331149b"}
Oct 30 20:52:53.367639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:53.367114    5002 scope.go:122] "RemoveContainer" containerID="7f4638d469dcfabe8a1fd0635d8aca55a59185a16e97396d2dda3353225b555b"
Oct 30 20:52:53.548578 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:53.548546    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"]
Oct 30 20:52:53.593442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:53.593409    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4"]
Oct 30 20:52:53.967771 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:53.967736    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1487eb-98e0-4f47-bd93-95ab47d1f626" path="/var/lib/kubelet/pods/5b1487eb-98e0-4f47-bd93-95ab47d1f626/volumes"
Oct 30 20:52:53.968149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:53.968135    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd640b3a-cea1-499f-a2fb-5e118f46407d" path="/var/lib/kubelet/pods/cd640b3a-cea1-499f-a2fb-5e118f46407d/volumes"
Oct 30 20:52:55.911793 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:55.911756    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:57.449295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.449263    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"]
Oct 30 20:52:57.449746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.449463    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" podUID="f19f2f77-1092-4e4a-8a29-31b779197202" containerName="discovery" containerID="cri-o://6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf" gracePeriod=30
Oct 30 20:52:57.756624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.756594    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:52:57.843754 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.843726    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-kube-api-access-jtpfz\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-kube-api-access-jtpfz\") pod \"f19f2f77-1092-4e4a-8a29-31b779197202\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") "
Oct 30 20:52:57.843930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.843841    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19f2f77-1092-4e4a-8a29-31b779197202-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19f2f77-1092-4e4a-8a29-31b779197202-local-certs\") pod \"f19f2f77-1092-4e4a-8a29-31b779197202\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") "
Oct 30 20:52:57.843930 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.843925    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-istio-token\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-istio-token\") pod \"f19f2f77-1092-4e4a-8a29-31b779197202\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") "
Oct 30 20:52:57.844056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.843972    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-dns-cert\") pod \"f19f2f77-1092-4e4a-8a29-31b779197202\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") "
Oct 30 20:52:57.844056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.844036    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-kubeconfig\") pod \"f19f2f77-1092-4e4a-8a29-31b779197202\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") "
Oct 30 20:52:57.844161 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.844076    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-ca-configmap\") pod \"f19f2f77-1092-4e4a-8a29-31b779197202\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") "
Oct 30 20:52:57.844161 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.844120    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-cacerts\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-cacerts\") pod \"f19f2f77-1092-4e4a-8a29-31b779197202\" (UID: \"f19f2f77-1092-4e4a-8a29-31b779197202\") "
Oct 30 20:52:57.846122 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.846076    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-ca-configmap" pod "f19f2f77-1092-4e4a-8a29-31b779197202" (UID: "f19f2f77-1092-4e4a-8a29-31b779197202"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:57.848573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.847065    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-dns-cert" pod "f19f2f77-1092-4e4a-8a29-31b779197202" (UID: "f19f2f77-1092-4e4a-8a29-31b779197202"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:57.848573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.847094    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-istio-token" pod "f19f2f77-1092-4e4a-8a29-31b779197202" (UID: "f19f2f77-1092-4e4a-8a29-31b779197202"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:57.848573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.847310    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-cacerts" pod "f19f2f77-1092-4e4a-8a29-31b779197202" (UID: "f19f2f77-1092-4e4a-8a29-31b779197202"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:57.848573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.847847    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19f2f77-1092-4e4a-8a29-31b779197202-local-certs" pod "f19f2f77-1092-4e4a-8a29-31b779197202" (UID: "f19f2f77-1092-4e4a-8a29-31b779197202"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:57.848573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.848364    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-kube-api-access-jtpfz" pod "f19f2f77-1092-4e4a-8a29-31b779197202" (UID: "f19f2f77-1092-4e4a-8a29-31b779197202"). InnerVolumeSpecName "kube-api-access-jtpfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:57.848573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.848373    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-kubeconfig" pod "f19f2f77-1092-4e4a-8a29-31b779197202" (UID: "f19f2f77-1092-4e4a-8a29-31b779197202"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:57.946031 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.945995    5002 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19f2f77-1092-4e4a-8a29-31b779197202-local-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:57.946031 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.946030    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:57.946219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.946045    5002 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-dns-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:57.946219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.946060    5002 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-istio-kubeconfig\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:57.946219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.946075    5002 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f19f2f77-1092-4e4a-8a29-31b779197202-istio-csr-ca-configmap\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:57.946219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.946089    5002 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f19f2f77-1092-4e4a-8a29-31b779197202-cacerts\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:57.946219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:57.946102    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpfz\" (UniqueName: \"kubernetes.io/projected/f19f2f77-1092-4e4a-8a29-31b779197202-kube-api-access-jtpfz\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:58.407396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.407298    5002 generic.go:356] "Generic (PLEG): container finished" podID="f19f2f77-1092-4e4a-8a29-31b779197202" containerID="6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf" exitCode=0
Oct 30 20:52:58.407396 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.407367    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"
Oct 30 20:52:58.407683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.407393    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" event={"ID":"f19f2f77-1092-4e4a-8a29-31b779197202","Type":"ContainerDied","Data":"6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf"}
Oct 30 20:52:58.407683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.407456    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq" event={"ID":"f19f2f77-1092-4e4a-8a29-31b779197202","Type":"ContainerDied","Data":"e32826f05aa49bae74e5db046827043621d160caa547835104d74f90b39509c2"}
Oct 30 20:52:58.407683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.407483    5002 scope.go:122] "RemoveContainer" containerID="6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf"
Oct 30 20:52:58.418710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.418690    5002 scope.go:122] "RemoveContainer" containerID="6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf"
Oct 30 20:52:58.418993 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:52:58.418963    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf\": container with ID starting with 6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf not found: ID does not exist" containerID="6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf"
Oct 30 20:52:58.419060 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.418989    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf"} err="failed to get container status \"6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf\": rpc error: code = NotFound desc = could not find container \"6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf\": container with ID starting with 6023644c17bfd3979fe255d846a604964d1320d6454964bceffa88e7567f0fcf not found: ID does not exist"
Oct 30 20:52:58.580337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.580312    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"]
Oct 30 20:52:58.630913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:58.630883    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq"]
Oct 30 20:52:59.967649 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:52:59.967617    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19f2f77-1092-4e4a-8a29-31b779197202" path="/var/lib/kubelet/pods/f19f2f77-1092-4e4a-8a29-31b779197202/volumes"
Oct 30 20:53:00.823430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:00.823390    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:05.989588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:05.987109    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:07.116192 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:07.116153    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:53:11.024940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:11.024880    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:12.608900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:12.608817    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"]
Oct 30 20:53:12.609334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:12.609039    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" containerID="cri-o://6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54" gracePeriod=30
Oct 30 20:53:14.559478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:14.559381    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:14.748872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:14.748838    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:15.794383 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:15.794354    5002 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:53:15.794831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:15.794411    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:53:16.227730 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:16.227154    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:17.892050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:17.891997    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:53:17.999805 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:17.999777    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-podinfo\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:17.999967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:17.999809    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-credential-socket\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:17.999967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:17.999842    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b08158-f394-4fed-84cf-9a3e0a476d3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b08158-f394-4fed-84cf-9a3e0a476d3e-istiod-ca-cert\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:17.999967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:17.999878    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-kube-api-access-n7v45\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-kube-api-access-n7v45\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:17.999967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:17.999901    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-socket\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:17.999967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:17.999925    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-data\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:18.000237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000023    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-token\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-token\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:18.000237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000039    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-credential-socket" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:18.000237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000117    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-certs\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:18.000237 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000218    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-envoy\") pod \"39b08158-f394-4fed-84cf-9a3e0a476d3e\" (UID: \"39b08158-f394-4fed-84cf-9a3e0a476d3e\") "
Oct 30 20:53:18.000434 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000271    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-data" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:18.000434 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000295    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-socket" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:18.000434 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000336    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b08158-f394-4fed-84cf-9a3e0a476d3e-istiod-ca-cert" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:18.000630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000519    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-certs" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:18.000768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000749    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.000835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000769    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.000835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000777    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b08158-f394-4fed-84cf-9a3e0a476d3e-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.000835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000785    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.000835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.000798    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.002430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.002403    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-envoy" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:18.002544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.002492    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-kube-api-access-n7v45" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "kube-api-access-n7v45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:18.002708 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.002692    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-podinfo" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:18.002770 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.002714    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-token" pod "39b08158-f394-4fed-84cf-9a3e0a476d3e" (UID: "39b08158-f394-4fed-84cf-9a3e0a476d3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:18.102028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.101998    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.102028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.102030    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.102242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.102046    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7v45\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-kube-api-access-n7v45\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.102242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.102060    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b08158-f394-4fed-84cf-9a3e0a476d3e-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:18.570489 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.570452    5002 generic.go:356] "Generic (PLEG): container finished" podID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerID="6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54" exitCode=0
Oct 30 20:53:18.570728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.570514    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"
Oct 30 20:53:18.570728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.570569    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" event={"ID":"39b08158-f394-4fed-84cf-9a3e0a476d3e","Type":"ContainerDied","Data":"6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54"}
Oct 30 20:53:18.570728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.570637    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp" event={"ID":"39b08158-f394-4fed-84cf-9a3e0a476d3e","Type":"ContainerDied","Data":"b7fd9125eb6c553d430eeedf7272f8eadf1648180f17c78bdb01bdcf990357bb"}
Oct 30 20:53:18.570728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.570667    5002 scope.go:122] "RemoveContainer" containerID="6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54"
Oct 30 20:53:18.585552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.585502    5002 scope.go:122] "RemoveContainer" containerID="6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54"
Oct 30 20:53:18.586014 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:18.585985    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54\": container with ID starting with 6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54 not found: ID does not exist" containerID="6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54"
Oct 30 20:53:18.586124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.586019    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54"} err="failed to get container status \"6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54\": rpc error: code = NotFound desc = could not find container \"6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54\": container with ID starting with 6b35b25a80ab7c05ac593282f3005d4a7e6863740101e33760cf796e8a6b2c54 not found: ID does not exist"
Oct 30 20:53:18.742912 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.742882    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"]
Oct 30 20:53:18.790774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:18.790752    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp"]
Oct 30 20:53:19.968459 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:19.968430    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" path="/var/lib/kubelet/pods/39b08158-f394-4fed-84cf-9a3e0a476d3e/volumes"
Oct 30 20:53:21.356442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:21.356405    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:25.894341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.894310    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"]
Oct 30 20:53:25.895005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.894988    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1e3ed774-c246-488d-828b-0bb4126a9fb0" containerName="payload-pre-processing"
Oct 30 20:53:25.895058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895005    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3ed774-c246-488d-828b-0bb4126a9fb0" containerName="payload-pre-processing"
Oct 30 20:53:25.895058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895019    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cd640b3a-cea1-499f-a2fb-5e118f46407d" containerName="payload-pre-processing"
Oct 30 20:53:25.895058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895023    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd640b3a-cea1-499f-a2fb-5e118f46407d" containerName="payload-pre-processing"
Oct 30 20:53:25.895058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895032    5002 cpu_manager.go:414] "containerMap: removing container" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy"
Oct 30 20:53:25.895058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895036    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy"
Oct 30 20:53:25.895058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895053    5002 cpu_manager.go:414] "containerMap: removing container" podUID="62aad480-43f9-45dd-8f51-2924f7091faf" containerName="payload-processing"
Oct 30 20:53:25.895058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895057    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62aad480-43f9-45dd-8f51-2924f7091faf" containerName="payload-processing"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895066    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895071    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895083    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5b1487eb-98e0-4f47-bd93-95ab47d1f626" containerName="payload-processing"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895088    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1487eb-98e0-4f47-bd93-95ab47d1f626" containerName="payload-processing"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895098    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f19f2f77-1092-4e4a-8a29-31b779197202" containerName="discovery"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895102    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19f2f77-1092-4e4a-8a29-31b779197202" containerName="discovery"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895109    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895113    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895209    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1487eb-98e0-4f47-bd93-95ab47d1f626" containerName="payload-processing"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895218    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19f2f77-1092-4e4a-8a29-31b779197202" containerName="discovery"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895225    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c790f12-d632-4162-85f1-8222c41b4d3f" containerName="istio-proxy"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895235    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3ed774-c246-488d-828b-0bb4126a9fb0" containerName="payload-pre-processing"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895241    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd640b3a-cea1-499f-a2fb-5e118f46407d" containerName="payload-pre-processing"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895247    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b08158-f394-4fed-84cf-9a3e0a476d3e" containerName="istio-proxy"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895255    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="62aad480-43f9-45dd-8f51-2924f7091faf" containerName="payload-processing"
Oct 30 20:53:25.895340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.895262    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9823d53-c5da-406a-a02c-1509bf0d2edc" containerName="istio-proxy"
Oct 30 20:53:25.898917 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.898901    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:25.957642 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:25.957610    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"]
Oct 30 20:53:26.002264 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.002230    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3f0723-serving-cert\""
Oct 30 20:53:26.005862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.005836    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8f95c6-f0fa-44c5-a034-05450655a684-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.006016 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.005893    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.006092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.006056    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-metrics-certs\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.006152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.006097    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhwk\" (UniqueName: \"kubernetes.io/projected/3e8f95c6-f0fa-44c5-a034-05450655a684-kube-api-access-hnhwk\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.107118 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.107086    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-metrics-certs\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.107118 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.107121    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhwk\" (UniqueName: \"kubernetes.io/projected/3e8f95c6-f0fa-44c5-a034-05450655a684-kube-api-access-hnhwk\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.107366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.107323    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8f95c6-f0fa-44c5-a034-05450655a684-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.107423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.107409    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.107581 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:26.107562    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-serving-cert: secret "maas-api-e2e-api-a-3f0723-serving-cert" not found
Oct 30 20:53:26.107689 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:26.107628    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls podName:3e8f95c6-f0fa-44c5-a034-05450655a684 nodeName:}" failed. No retries permitted until 2026-10-30 20:53:26.607610307 +0000 UTC m=+2032.756661289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls") pod "maas-api-e2e-api-a-3f0723-85795bb986-88nb6" (UID: "3e8f95c6-f0fa-44c5-a034-05450655a684") : secret "maas-api-e2e-api-a-3f0723-serving-cert" not found
Oct 30 20:53:26.107954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.107935    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8f95c6-f0fa-44c5-a034-05450655a684-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.109561 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.109522    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-metrics-certs\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.174361 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.174216    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhwk\" (UniqueName: \"kubernetes.io/projected/3e8f95c6-f0fa-44c5-a034-05450655a684-kube-api-access-hnhwk\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.175121 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.175088    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:26.287817 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.287785    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:53:26.363925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.363894    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:26.550498 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.550467    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:26.616127 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.616103    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.618624 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.618603    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls\") pod \"maas-api-e2e-api-a-3f0723-85795bb986-88nb6\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:26.813171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:26.813109    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:27.195052 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:53:27.195020    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8f95c6_f0fa_44c5_a034_05450655a684.slice/crio-facb2bb1a2471282ceeb3dd58addf59a662150fefc712d82990341c29f812950 WatchSource:0}: Error finding container facb2bb1a2471282ceeb3dd58addf59a662150fefc712d82990341c29f812950: Status 404 returned error can't find the container with id facb2bb1a2471282ceeb3dd58addf59a662150fefc712d82990341c29f812950
Oct 30 20:53:27.195402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:27.195075    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"]
Oct 30 20:53:27.196657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:27.196640    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:53:27.638626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:27.638597    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6" event={"ID":"3e8f95c6-f0fa-44c5-a034-05450655a684","Type":"ContainerStarted","Data":"facb2bb1a2471282ceeb3dd58addf59a662150fefc712d82990341c29f812950"}
Oct 30 20:53:28.648053 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:28.648022    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6" event={"ID":"3e8f95c6-f0fa-44c5-a034-05450655a684","Type":"ContainerStarted","Data":"ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6"}
Oct 30 20:53:28.648422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:28.648157    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:28.785619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:28.785563    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6" podStartSLOduration=2.994290623 podStartE2EDuration="3.785550575s" totalImagesPullingTime="791.259952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:28.776351569 +0000 UTC m=+2034.925402556" watchObservedRunningTime="2026-10-30 20:53:28.785550575 +0000 UTC m=+2034.934601561"
Oct 30 20:53:31.095318 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:31.095262    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:34.660836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:34.660813    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:53:35.460698 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:35.460666    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:35.640716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:35.640685    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:36.394826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:36.394600    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"]
Oct 30 20:53:36.801156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:36.801129    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:36.986304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:36.986266    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:37.116333 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:37.116255    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:53:37.177710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:37.177681    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:37.514918 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:37.514866    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:37.705140 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:37.705111    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:38.259244 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:38.259213    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:41.397669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.397631    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws" podUID="fca7b628-a8be-417d-a102-b2a644603d16" containerName="payload-processing" containerID="cri-o://79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7" gracePeriod=25
Oct 30 20:53:41.685754 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.685732    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:53:41.757858 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.757831    5002 generic.go:356] "Generic (PLEG): container finished" podID="fca7b628-a8be-417d-a102-b2a644603d16" containerID="79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7" exitCode=0
Oct 30 20:53:41.757999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.757874    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"
Oct 30 20:53:41.757999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.757916    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws" event={"ID":"fca7b628-a8be-417d-a102-b2a644603d16","Type":"ContainerDied","Data":"79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7"}
Oct 30 20:53:41.757999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.757978    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws" event={"ID":"fca7b628-a8be-417d-a102-b2a644603d16","Type":"ContainerDied","Data":"4a8a72b62c67768a6e0257cf45838a5b8f0c643dba5f0480b76b6b33d174f52b"}
Oct 30 20:53:41.758157 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.758005    5002 scope.go:122] "RemoveContainer" containerID="79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7"
Oct 30 20:53:41.768156 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.768131    5002 scope.go:122] "RemoveContainer" containerID="79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7"
Oct 30 20:53:41.768413 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:41.768393    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7\": container with ID starting with 79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7 not found: ID does not exist" containerID="79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7"
Oct 30 20:53:41.768466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.768420    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7"} err="failed to get container status \"79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7\": rpc error: code = NotFound desc = could not find container \"79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7\": container with ID starting with 79693bdbcf17b7e0084e044de3fecefec3d34adfad9acdec2fbf1c69a2eb19b7 not found: ID does not exist"
Oct 30 20:53:41.892670 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.892639    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca7b628-a8be-417d-a102-b2a644603d16-kube-api-access-zxr2m\" (UniqueName: \"kubernetes.io/projected/fca7b628-a8be-417d-a102-b2a644603d16-kube-api-access-zxr2m\") pod \"fca7b628-a8be-417d-a102-b2a644603d16\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") "
Oct 30 20:53:41.893106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.893083    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca7b628-a8be-417d-a102-b2a644603d16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b628-a8be-417d-a102-b2a644603d16-plugins-config-volume\") pod \"fca7b628-a8be-417d-a102-b2a644603d16\" (UID: \"fca7b628-a8be-417d-a102-b2a644603d16\") "
Oct 30 20:53:41.898362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.895148    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca7b628-a8be-417d-a102-b2a644603d16-plugins-config-volume" pod "fca7b628-a8be-417d-a102-b2a644603d16" (UID: "fca7b628-a8be-417d-a102-b2a644603d16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:41.923795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.922368    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"]
Oct 30 20:53:41.923795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.922707    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr" podUID="f6883e6c-3852-4858-9724-8e523a3783d2" containerName="maas-api" containerID="cri-o://e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719" gracePeriod=30
Oct 30 20:53:41.947779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.947749    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca7b628-a8be-417d-a102-b2a644603d16-kube-api-access-zxr2m" pod "fca7b628-a8be-417d-a102-b2a644603d16" (UID: "fca7b628-a8be-417d-a102-b2a644603d16"). InnerVolumeSpecName "kube-api-access-zxr2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:41.999575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.999547    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca7b628-a8be-417d-a102-b2a644603d16-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:41.999702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:41.999579    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxr2m\" (UniqueName: \"kubernetes.io/projected/fca7b628-a8be-417d-a102-b2a644603d16-kube-api-access-zxr2m\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:42.041282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.041257    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:42.183950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.183925    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:53:42.189769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.189745    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"]
Oct 30 20:53:42.195732 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.195710    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws"]
Oct 30 20:53:42.303280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.303241    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls\") pod \"f6883e6c-3852-4858-9724-8e523a3783d2\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") "
Oct 30 20:53:42.303662 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.303637    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-metrics-certs\") pod \"f6883e6c-3852-4858-9724-8e523a3783d2\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") "
Oct 30 20:53:42.303878 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.303854    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6883e6c-3852-4858-9724-8e523a3783d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6883e6c-3852-4858-9724-8e523a3783d2-openshift-service-ca\") pod \"f6883e6c-3852-4858-9724-8e523a3783d2\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") "
Oct 30 20:53:42.304001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.303947    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6883e6c-3852-4858-9724-8e523a3783d2-kube-api-access-7vk9s\" (UniqueName: \"kubernetes.io/projected/f6883e6c-3852-4858-9724-8e523a3783d2-kube-api-access-7vk9s\") pod \"f6883e6c-3852-4858-9724-8e523a3783d2\" (UID: \"f6883e6c-3852-4858-9724-8e523a3783d2\") "
Oct 30 20:53:42.304572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.304547    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6883e6c-3852-4858-9724-8e523a3783d2-openshift-service-ca" pod "f6883e6c-3852-4858-9724-8e523a3783d2" (UID: "f6883e6c-3852-4858-9724-8e523a3783d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:42.306524 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.306501    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6883e6c-3852-4858-9724-8e523a3783d2-kube-api-access-7vk9s" pod "f6883e6c-3852-4858-9724-8e523a3783d2" (UID: "f6883e6c-3852-4858-9724-8e523a3783d2"). InnerVolumeSpecName "kube-api-access-7vk9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:42.306871 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.306854    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-metrics-certs" pod "f6883e6c-3852-4858-9724-8e523a3783d2" (UID: "f6883e6c-3852-4858-9724-8e523a3783d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:42.309423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.309398    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls" pod "f6883e6c-3852-4858-9724-8e523a3783d2" (UID: "f6883e6c-3852-4858-9724-8e523a3783d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:42.405464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.405422    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:42.405464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.405464    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6883e6c-3852-4858-9724-8e523a3783d2-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:42.405464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.405475    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6883e6c-3852-4858-9724-8e523a3783d2-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:42.405900 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.405484    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vk9s\" (UniqueName: \"kubernetes.io/projected/f6883e6c-3852-4858-9724-8e523a3783d2-kube-api-access-7vk9s\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:42.767141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.767108    5002 generic.go:356] "Generic (PLEG): container finished" podID="f6883e6c-3852-4858-9724-8e523a3783d2" containerID="e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719" exitCode=0
Oct 30 20:53:42.767315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.767151    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"
Oct 30 20:53:42.767315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.767197    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr" event={"ID":"f6883e6c-3852-4858-9724-8e523a3783d2","Type":"ContainerDied","Data":"e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719"}
Oct 30 20:53:42.767315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.767251    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr" event={"ID":"f6883e6c-3852-4858-9724-8e523a3783d2","Type":"ContainerDied","Data":"d98bb0d57215ec58fcb6a988920bcb64894002aac38cd815b3f26930af2feebf"}
Oct 30 20:53:42.767315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.767276    5002 scope.go:122] "RemoveContainer" containerID="e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719"
Oct 30 20:53:42.778827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.778807    5002 scope.go:122] "RemoveContainer" containerID="e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719"
Oct 30 20:53:42.779106 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:42.779089    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719\": container with ID starting with e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719 not found: ID does not exist" containerID="e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719"
Oct 30 20:53:42.779175 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.779113    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719"} err="failed to get container status \"e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719\": rpc error: code = NotFound desc = could not find container \"e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719\": container with ID starting with e1cc79b95a74982f89246c5369c96e00f72e0bc5e0a7552c3f3378c017bd0719 not found: ID does not exist"
Oct 30 20:53:42.969795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.969744    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"]
Oct 30 20:53:42.992975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:42.992944    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-99c030-6dbf9fd4f6-qdnwr"]
Oct 30 20:53:43.873843 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:43.873813    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:43.967356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:43.967328    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6883e6c-3852-4858-9724-8e523a3783d2" path="/var/lib/kubelet/pods/f6883e6c-3852-4858-9724-8e523a3783d2/volumes"
Oct 30 20:53:43.967743 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:43.967728    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca7b628-a8be-417d-a102-b2a644603d16" path="/var/lib/kubelet/pods/fca7b628-a8be-417d-a102-b2a644603d16/volumes"
Oct 30 20:53:44.105415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:44.105368    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:44.677380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:44.677349    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:44.952231 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:44.952145    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:45.231776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:45.231731    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:45.444455 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:45.444420    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"]
Oct 30 20:53:45.444714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:45.444684    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy" containerID="cri-o://f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd" gracePeriod=30
Oct 30 20:53:45.756380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:45.756347    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:48.420367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:48.420334    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"]
Oct 30 20:53:48.527544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:48.527492    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"]
Oct 30 20:53:48.801794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:48.801769    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:48.987928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:48.987896    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:49.336042 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:49.336012    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:50.727436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.727411    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:53:50.811168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811142    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-kube-api-access-sv4qj\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-kube-api-access-sv4qj\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811185    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-data\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811369 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811331    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istiod-ca-cert\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811372    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-credential-socket\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811405    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-token\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811468    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-socket\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811515    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-certs\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811514    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-data" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:50.811707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811572    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-envoy\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811696    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-podinfo\") pod \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\" (UID: \"d5a71733-90b6-46c8-8c95-80e0ed2d9251\") "
Oct 30 20:53:50.811808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811755    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-socket" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:50.811993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.811967    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-certs" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:50.812195 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.812172    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istiod-ca-cert" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:50.812266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.812202    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.812266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.812225    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.812266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.812238    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.812389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.812259    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-credential-socket" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:50.814437 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.814417    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-token" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:50.814567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.814451    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-envoy" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:50.814729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.814710    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-podinfo" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:50.815230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.815206    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-kube-api-access-sv4qj" pod "d5a71733-90b6-46c8-8c95-80e0ed2d9251" (UID: "d5a71733-90b6-46c8-8c95-80e0ed2d9251"). InnerVolumeSpecName "kube-api-access-sv4qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:50.870189 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.870161    5002 generic.go:356] "Generic (PLEG): container finished" podID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerID="f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd" exitCode=0
Oct 30 20:53:50.870311 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.870210    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"
Oct 30 20:53:50.870311 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.870252    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" event={"ID":"d5a71733-90b6-46c8-8c95-80e0ed2d9251","Type":"ContainerDied","Data":"f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd"}
Oct 30 20:53:50.870311 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.870305    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9" event={"ID":"d5a71733-90b6-46c8-8c95-80e0ed2d9251","Type":"ContainerDied","Data":"43e3775db18abbe892a9b727da62c8eecc1d10533cb9c67ba9ee293601d10e89"}
Oct 30 20:53:50.870441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.870327    5002 scope.go:122] "RemoveContainer" containerID="f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd"
Oct 30 20:53:50.884928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.884909    5002 scope.go:122] "RemoveContainer" containerID="51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f"
Oct 30 20:53:50.894068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.894053    5002 scope.go:122] "RemoveContainer" containerID="f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd"
Oct 30 20:53:50.894323 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:50.894305    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd\": container with ID starting with f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd not found: ID does not exist" containerID="f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd"
Oct 30 20:53:50.894388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.894336    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd"} err="failed to get container status \"f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd\": rpc error: code = NotFound desc = could not find container \"f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd\": container with ID starting with f85e6034a98ef11619595726e8bf7f7a916dc7bb094d5734b70ae0038c5757dd not found: ID does not exist"
Oct 30 20:53:50.894388 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.894375    5002 scope.go:122] "RemoveContainer" containerID="51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f"
Oct 30 20:53:50.894667 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:50.894651    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f\": container with ID starting with 51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f not found: ID does not exist" containerID="51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f"
Oct 30 20:53:50.894710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.894673    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f"} err="failed to get container status \"51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f\": rpc error: code = NotFound desc = could not find container \"51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f\": container with ID starting with 51036ba1c9593c763d27b3ac6f4037d9696721aaa110b0c01b76025fb963f67f not found: ID does not exist"
Oct 30 20:53:50.913366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.913260    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.913366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.913287    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.913366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.913302    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.913366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.913316    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.913366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.913328    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a71733-90b6-46c8-8c95-80e0ed2d9251-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.913366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.913342    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4qj\" (UniqueName: \"kubernetes.io/projected/d5a71733-90b6-46c8-8c95-80e0ed2d9251-kube-api-access-sv4qj\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.971286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:50.971224    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:51.027981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:51.027948    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"]
Oct 30 20:53:51.035606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:51.035581    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9"]
Oct 30 20:53:51.381158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:51.381073    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:51.612472 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:51.612436    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:51.967177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:51.967141    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" path="/var/lib/kubelet/pods/d5a71733-90b6-46c8-8c95-80e0ed2d9251/volumes"
Oct 30 20:53:53.420909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.420861    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4" podUID="3ff11802-8f1a-43ea-933c-96b271fc17eb" containerName="payload-processing" containerID="cri-o://1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9" gracePeriod=25
Oct 30 20:53:53.528714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.528660    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n" podUID="c71c15ce-1584-4d9b-a8ae-f172e383b939" containerName="payload-pre-processing" containerID="cri-o://f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc" gracePeriod=25
Oct 30 20:53:53.707951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.707926    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:53:53.826592 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.826570    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:53:53.856857 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.856822    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff11802-8f1a-43ea-933c-96b271fc17eb-kube-api-access-4sfwv\" (UniqueName: \"kubernetes.io/projected/3ff11802-8f1a-43ea-933c-96b271fc17eb-kube-api-access-4sfwv\") pod \"3ff11802-8f1a-43ea-933c-96b271fc17eb\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") "
Oct 30 20:53:53.857043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.856961    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff11802-8f1a-43ea-933c-96b271fc17eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff11802-8f1a-43ea-933c-96b271fc17eb-plugins-config-volume\") pod \"3ff11802-8f1a-43ea-933c-96b271fc17eb\" (UID: \"3ff11802-8f1a-43ea-933c-96b271fc17eb\") "
Oct 30 20:53:53.857643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.857615    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff11802-8f1a-43ea-933c-96b271fc17eb-plugins-config-volume" pod "3ff11802-8f1a-43ea-933c-96b271fc17eb" (UID: "3ff11802-8f1a-43ea-933c-96b271fc17eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:53.864672 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.864646    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff11802-8f1a-43ea-933c-96b271fc17eb-kube-api-access-4sfwv" pod "3ff11802-8f1a-43ea-933c-96b271fc17eb" (UID: "3ff11802-8f1a-43ea-933c-96b271fc17eb"). InnerVolumeSpecName "kube-api-access-4sfwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.885378 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.885357    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"]
Oct 30 20:53:53.885855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.885825    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s" podUID="2145966a-2cd2-458b-a7db-5020d14a4913" containerName="maas-api" containerID="cri-o://ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2" gracePeriod=30
Oct 30 20:53:53.907239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.907210    5002 generic.go:356] "Generic (PLEG): container finished" podID="3ff11802-8f1a-43ea-933c-96b271fc17eb" containerID="1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9" exitCode=0
Oct 30 20:53:53.907367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.907248    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"
Oct 30 20:53:53.907367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.907287    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4" event={"ID":"3ff11802-8f1a-43ea-933c-96b271fc17eb","Type":"ContainerDied","Data":"1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9"}
Oct 30 20:53:53.907367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.907336    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4" event={"ID":"3ff11802-8f1a-43ea-933c-96b271fc17eb","Type":"ContainerDied","Data":"877817192f3115c3376787c6a7a8ac9fa12bbeaa4c954f2035e2e3a2ae74b984"}
Oct 30 20:53:53.907367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.907362    5002 scope.go:122] "RemoveContainer" containerID="1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9"
Oct 30 20:53:53.912614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.909575    5002 generic.go:356] "Generic (PLEG): container finished" podID="c71c15ce-1584-4d9b-a8ae-f172e383b939" containerID="f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc" exitCode=0
Oct 30 20:53:53.912614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.909674    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n" event={"ID":"c71c15ce-1584-4d9b-a8ae-f172e383b939","Type":"ContainerDied","Data":"f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc"}
Oct 30 20:53:53.912614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.909714    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n" event={"ID":"c71c15ce-1584-4d9b-a8ae-f172e383b939","Type":"ContainerDied","Data":"240e57614166a23c165c726cdbe81e22970fad2b0fb394720ba3cbbc84947ac3"}
Oct 30 20:53:53.912614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.909785    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"
Oct 30 20:53:53.959583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.958052    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71c15ce-1584-4d9b-a8ae-f172e383b939-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71c15ce-1584-4d9b-a8ae-f172e383b939-plugins-config-volume\") pod \"c71c15ce-1584-4d9b-a8ae-f172e383b939\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") "
Oct 30 20:53:53.959583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.958160    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71c15ce-1584-4d9b-a8ae-f172e383b939-kube-api-access-zfsx7\" (UniqueName: \"kubernetes.io/projected/c71c15ce-1584-4d9b-a8ae-f172e383b939-kube-api-access-zfsx7\") pod \"c71c15ce-1584-4d9b-a8ae-f172e383b939\" (UID: \"c71c15ce-1584-4d9b-a8ae-f172e383b939\") "
Oct 30 20:53:53.959583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.958437    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71c15ce-1584-4d9b-a8ae-f172e383b939-plugins-config-volume" pod "c71c15ce-1584-4d9b-a8ae-f172e383b939" (UID: "c71c15ce-1584-4d9b-a8ae-f172e383b939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:53.959583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.958837    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71c15ce-1584-4d9b-a8ae-f172e383b939-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.959583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.958859    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfwv\" (UniqueName: \"kubernetes.io/projected/3ff11802-8f1a-43ea-933c-96b271fc17eb-kube-api-access-4sfwv\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.959583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.958872    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff11802-8f1a-43ea-933c-96b271fc17eb-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.963702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:53.963676    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71c15ce-1584-4d9b-a8ae-f172e383b939-kube-api-access-zfsx7" pod "c71c15ce-1584-4d9b-a8ae-f172e383b939" (UID: "c71c15ce-1584-4d9b-a8ae-f172e383b939"). InnerVolumeSpecName "kube-api-access-zfsx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:54.001031 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.001008    5002 scope.go:122] "RemoveContainer" containerID="1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9"
Oct 30 20:53:54.001389 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:54.001369    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9\": container with ID starting with 1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9 not found: ID does not exist" containerID="1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9"
Oct 30 20:53:54.001499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.001400    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9"} err="failed to get container status \"1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9\": rpc error: code = NotFound desc = could not find container \"1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9\": container with ID starting with 1912c1be1b4ceb5d5c068b6301347377a790c7ff5c2e6c9d8d8bbca42a27aac9 not found: ID does not exist"
Oct 30 20:53:54.001499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.001426    5002 scope.go:122] "RemoveContainer" containerID="f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc"
Oct 30 20:53:54.011973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.011936    5002 scope.go:122] "RemoveContainer" containerID="f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc"
Oct 30 20:53:54.012222 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:54.012195    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc\": container with ID starting with f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc not found: ID does not exist" containerID="f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc"
Oct 30 20:53:54.012334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.012229    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc"} err="failed to get container status \"f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc\": rpc error: code = NotFound desc = could not find container \"f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc\": container with ID starting with f00e123cf186958a9e94c1b69925e01083cd8c1d4c4bd614ad412e867a7f43fc not found: ID does not exist"
Oct 30 20:53:54.017759 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.017734    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:54.059842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.059811    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsx7\" (UniqueName: \"kubernetes.io/projected/c71c15ce-1584-4d9b-a8ae-f172e383b939-kube-api-access-zfsx7\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.070267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.068298    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"]
Oct 30 20:53:54.072973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.072939    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4"]
Oct 30 20:53:54.170257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.170231    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:53:54.262696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.262666    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2145966a-2cd2-458b-a7db-5020d14a4913-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2145966a-2cd2-458b-a7db-5020d14a4913-openshift-service-ca\") pod \"2145966a-2cd2-458b-a7db-5020d14a4913\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") "
Oct 30 20:53:54.262864 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.262727    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-metrics-certs\") pod \"2145966a-2cd2-458b-a7db-5020d14a4913\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") "
Oct 30 20:53:54.262864 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.262776    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2145966a-2cd2-458b-a7db-5020d14a4913-kube-api-access-7pwm9\" (UniqueName: \"kubernetes.io/projected/2145966a-2cd2-458b-a7db-5020d14a4913-kube-api-access-7pwm9\") pod \"2145966a-2cd2-458b-a7db-5020d14a4913\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") "
Oct 30 20:53:54.262864 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.262836    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls\") pod \"2145966a-2cd2-458b-a7db-5020d14a4913\" (UID: \"2145966a-2cd2-458b-a7db-5020d14a4913\") "
Oct 30 20:53:54.263117 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.263080    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2145966a-2cd2-458b-a7db-5020d14a4913-openshift-service-ca" pod "2145966a-2cd2-458b-a7db-5020d14a4913" (UID: "2145966a-2cd2-458b-a7db-5020d14a4913"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:54.263366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.263340    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2145966a-2cd2-458b-a7db-5020d14a4913-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.264925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.264905    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2145966a-2cd2-458b-a7db-5020d14a4913-kube-api-access-7pwm9" pod "2145966a-2cd2-458b-a7db-5020d14a4913" (UID: "2145966a-2cd2-458b-a7db-5020d14a4913"). InnerVolumeSpecName "kube-api-access-7pwm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:54.264925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.264908    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls" pod "2145966a-2cd2-458b-a7db-5020d14a4913" (UID: "2145966a-2cd2-458b-a7db-5020d14a4913"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:54.265041 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.264932    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-metrics-certs" pod "2145966a-2cd2-458b-a7db-5020d14a4913" (UID: "2145966a-2cd2-458b-a7db-5020d14a4913"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:54.364692 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.364660    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.364692 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.364689    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwm9\" (UniqueName: \"kubernetes.io/projected/2145966a-2cd2-458b-a7db-5020d14a4913-kube-api-access-7pwm9\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.364692 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.364701    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2145966a-2cd2-458b-a7db-5020d14a4913-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.492035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.492000    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"]
Oct 30 20:53:54.499079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.499048    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n"]
Oct 30 20:53:54.948803 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.948765    5002 generic.go:356] "Generic (PLEG): container finished" podID="2145966a-2cd2-458b-a7db-5020d14a4913" containerID="ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2" exitCode=0
Oct 30 20:53:54.948982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.948920    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"
Oct 30 20:53:54.949879 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.949853    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s" event={"ID":"2145966a-2cd2-458b-a7db-5020d14a4913","Type":"ContainerDied","Data":"ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2"}
Oct 30 20:53:54.949947 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.949913    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s" event={"ID":"2145966a-2cd2-458b-a7db-5020d14a4913","Type":"ContainerDied","Data":"cf4963e94cd337a4885438f9ba04746641cd289cf7dd6b848c5aa8eb17b0df39"}
Oct 30 20:53:54.949947 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.949942    5002 scope.go:122] "RemoveContainer" containerID="ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2"
Oct 30 20:53:54.964023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.963999    5002 scope.go:122] "RemoveContainer" containerID="ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2"
Oct 30 20:53:54.964307 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:53:54.964288    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2\": container with ID starting with ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2 not found: ID does not exist" containerID="ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2"
Oct 30 20:53:54.964381 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:54.964317    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2"} err="failed to get container status \"ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2\": rpc error: code = NotFound desc = could not find container \"ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2\": container with ID starting with ad5a049bdaaf609cb50c3fcaa3fee5c83938bb8fea44493412e9362a6ccb6fc2 not found: ID does not exist"
Oct 30 20:53:55.117842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:55.117807    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"]
Oct 30 20:53:55.121196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:55.121176    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-64l6s"]
Oct 30 20:53:55.611227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:55.611192    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:55.847507 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:55.847471    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:55.969514 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:55.969472    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2145966a-2cd2-458b-a7db-5020d14a4913" path="/var/lib/kubelet/pods/2145966a-2cd2-458b-a7db-5020d14a4913/volumes"
Oct 30 20:53:55.970096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:55.970072    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff11802-8f1a-43ea-933c-96b271fc17eb" path="/var/lib/kubelet/pods/3ff11802-8f1a-43ea-933c-96b271fc17eb/volumes"
Oct 30 20:53:55.970547 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:55.970513    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71c15ce-1584-4d9b-a8ae-f172e383b939" path="/var/lib/kubelet/pods/c71c15ce-1584-4d9b-a8ae-f172e383b939/volumes"
Oct 30 20:53:56.054058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:56.054033    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:57.723144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:57.723106    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"]
Oct 30 20:53:58.138475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:58.138395    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:58.332402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:53:58.332377    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:00.316968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:00.316936    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:01.203705 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:01.203633    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:02.724356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:02.724289    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp" podUID="8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" containerName="payload-processing" containerID="cri-o://6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765" gracePeriod=25
Oct 30 20:54:02.991962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:02.991908    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:54:03.014827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.014795    5002 generic.go:356] "Generic (PLEG): container finished" podID="8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" containerID="6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765" exitCode=0
Oct 30 20:54:03.014944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.014853    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"
Oct 30 20:54:03.014944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.014873    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp" event={"ID":"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2","Type":"ContainerDied","Data":"6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765"}
Oct 30 20:54:03.014944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.014918    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp" event={"ID":"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2","Type":"ContainerDied","Data":"57d682cb6ec4c2b74dd325e61c6f2fec41cd8b44d1ad7fe3d857aeba360bf8a3"}
Oct 30 20:54:03.014944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.014939    5002 scope.go:122] "RemoveContainer" containerID="6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765"
Oct 30 20:54:03.026882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.026857    5002 scope.go:122] "RemoveContainer" containerID="6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765"
Oct 30 20:54:03.027169 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:54:03.027145    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765\": container with ID starting with 6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765 not found: ID does not exist" containerID="6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765"
Oct 30 20:54:03.027234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.027186    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765"} err="failed to get container status \"6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765\": rpc error: code = NotFound desc = could not find container \"6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765\": container with ID starting with 6cb431fbd4a632a974673e2cea1019efbdd065134c623bb2b8ad653762a19765 not found: ID does not exist"
Oct 30 20:54:03.073496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.073474    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-kube-api-access-sbnrl\" (UniqueName: \"kubernetes.io/projected/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-kube-api-access-sbnrl\") pod \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") "
Oct 30 20:54:03.073760 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.073748    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-plugins-config-volume\") pod \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\" (UID: \"8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2\") "
Oct 30 20:54:03.074087 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.074063    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-plugins-config-volume" pod "8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" (UID: "8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:03.075584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.075566    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-kube-api-access-sbnrl" pod "8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" (UID: "8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2"). InnerVolumeSpecName "kube-api-access-sbnrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:03.175007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.174977    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnrl\" (UniqueName: \"kubernetes.io/projected/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-kube-api-access-sbnrl\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:03.175007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.175002    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:03.436984 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.436952    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"]
Oct 30 20:54:03.441736 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.441711    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp"]
Oct 30 20:54:03.967219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:03.967188    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" path="/var/lib/kubelet/pods/8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2/volumes"
Oct 30 20:54:04.073804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:04.073773    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:04.268525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:04.268444    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:06.445120 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:06.445080    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:07.117654 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:54:07.117615    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:54:07.747297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:07.747262    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:07.937286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:07.937232    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:08.112909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.112814    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"]
Oct 30 20:54:08.113426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113409    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2145966a-2cd2-458b-a7db-5020d14a4913" containerName="maas-api"
Oct 30 20:54:08.113426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113424    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2145966a-2cd2-458b-a7db-5020d14a4913" containerName="maas-api"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113453    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" containerName="payload-processing"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113458    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" containerName="payload-processing"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113465    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f6883e6c-3852-4858-9724-8e523a3783d2" containerName="maas-api"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113470    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6883e6c-3852-4858-9724-8e523a3783d2" containerName="maas-api"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113480    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113484    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113492    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fca7b628-a8be-417d-a102-b2a644603d16" containerName="payload-processing"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113496    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca7b628-a8be-417d-a102-b2a644603d16" containerName="payload-processing"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113503    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3ff11802-8f1a-43ea-933c-96b271fc17eb" containerName="payload-processing"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113508    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff11802-8f1a-43ea-933c-96b271fc17eb" containerName="payload-processing"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113517    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113521    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113548    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c71c15ce-1584-4d9b-a8ae-f172e383b939" containerName="payload-pre-processing"
Oct 30 20:54:08.113619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113553    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71c15ce-1584-4d9b-a8ae-f172e383b939" containerName="payload-pre-processing"
Oct 30 20:54:08.114286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113648    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy"
Oct 30 20:54:08.114286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113656    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71c15ce-1584-4d9b-a8ae-f172e383b939" containerName="payload-pre-processing"
Oct 30 20:54:08.114286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113666    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2145966a-2cd2-458b-a7db-5020d14a4913" containerName="maas-api"
Oct 30 20:54:08.114286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113676    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da68d8f-12b1-48dd-8e1c-81cdb0db1ab2" containerName="payload-processing"
Oct 30 20:54:08.114286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113682    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6883e6c-3852-4858-9724-8e523a3783d2" containerName="maas-api"
Oct 30 20:54:08.114286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113688    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca7b628-a8be-417d-a102-b2a644603d16" containerName="payload-processing"
Oct 30 20:54:08.114286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.113694    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff11802-8f1a-43ea-933c-96b271fc17eb" containerName="payload-processing"
Oct 30 20:54:08.118454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.118431    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.174381 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.174349    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"]
Oct 30 20:54:08.227028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.226849    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-27219c73\""
Oct 30 20:54:08.227028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.226995    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-27219c73-dockercfg-55d77\""
Oct 30 20:54:08.258071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.257378    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhvp\" (UniqueName: \"kubernetes.io/projected/73f8acac-b739-48ad-acfd-67e863e35bc0-kube-api-access-kqhvp\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.258071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.257560    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8acac-b739-48ad-acfd-67e863e35bc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.284188 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.284112    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"]
Oct 30 20:54:08.285631 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.285609    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a71733-90b6-46c8-8c95-80e0ed2d9251" containerName="istio-proxy"
Oct 30 20:54:08.289559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.289526    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:08.311229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.311203    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"]
Oct 30 20:54:08.359346 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.359313    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhvp\" (UniqueName: \"kubernetes.io/projected/73f8acac-b739-48ad-acfd-67e863e35bc0-kube-api-access-kqhvp\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.359510 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.359490    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8acac-b739-48ad-acfd-67e863e35bc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.360351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.360331    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8acac-b739-48ad-acfd-67e863e35bc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.417134 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.417078    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhvp\" (UniqueName: \"kubernetes.io/projected/73f8acac-b739-48ad-acfd-67e863e35bc0-kube-api-access-kqhvp\") pod \"payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.435933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.435912    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:08.460826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.460741    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5qw\" (UniqueName: \"kubernetes.io/projected/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-kube-api-access-qv5qw\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-drklv\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:08.460956 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.460899    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-drklv\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:08.561849 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.561812    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5qw\" (UniqueName: \"kubernetes.io/projected/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-kube-api-access-qv5qw\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-drklv\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:08.562030 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.561928    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-drklv\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:08.562894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.562867    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-drklv\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:08.619568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.619545    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5qw\" (UniqueName: \"kubernetes.io/projected/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-kube-api-access-qv5qw\") pod \"payload-processing-e2e-mt-27219c73-57f8548544-drklv\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") " pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:08.811800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.811776    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"]
Oct 30 20:54:08.814572 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:54:08.814514    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f8acac_b739_48ad_acfd_67e863e35bc0.slice/crio-17f0fb31ae0c227ed9825c9d730aae824ef163c67ec14b758462f685186544f5 WatchSource:0}: Error finding container 17f0fb31ae0c227ed9825c9d730aae824ef163c67ec14b758462f685186544f5: Status 404 returned error can't find the container with id 17f0fb31ae0c227ed9825c9d730aae824ef163c67ec14b758462f685186544f5
Oct 30 20:54:08.904182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:08.904161    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:09.073355 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:09.073280    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7" event={"ID":"73f8acac-b739-48ad-acfd-67e863e35bc0","Type":"ContainerStarted","Data":"f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441"}
Oct 30 20:54:09.073355 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:09.073337    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7" event={"ID":"73f8acac-b739-48ad-acfd-67e863e35bc0","Type":"ContainerStarted","Data":"17f0fb31ae0c227ed9825c9d730aae824ef163c67ec14b758462f685186544f5"}
Oct 30 20:54:09.073565 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:09.073367    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:09.192625 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:09.192575    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7" podStartSLOduration=1.192561293 podStartE2EDuration="1.192561293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:09.189508948 +0000 UTC m=+2075.338559935" watchObservedRunningTime="2026-10-30 20:54:09.192561293 +0000 UTC m=+2075.341612273"
Oct 30 20:54:09.284485 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:54:09.284442    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe21dcb_a1a0_4c41_8f9e_b2dc555cd9bb.slice/crio-c7d11c82154c585ec495dfd539101e3bd2522bfd4ececa2a86ad683a7621a1ed WatchSource:0}: Error finding container c7d11c82154c585ec495dfd539101e3bd2522bfd4ececa2a86ad683a7621a1ed: Status 404 returned error can't find the container with id c7d11c82154c585ec495dfd539101e3bd2522bfd4ececa2a86ad683a7621a1ed
Oct 30 20:54:09.284485 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:09.284449    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"]
Oct 30 20:54:10.082104 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:10.082067    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv" event={"ID":"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb","Type":"ContainerStarted","Data":"69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d"}
Oct 30 20:54:10.082478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:10.082126    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv" event={"ID":"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb","Type":"ContainerStarted","Data":"c7d11c82154c585ec495dfd539101e3bd2522bfd4ececa2a86ad683a7621a1ed"}
Oct 30 20:54:10.082478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:10.082286    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:10.202113 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:10.202067    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv" podStartSLOduration=2.202054241 podStartE2EDuration="2.202054241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:10.198829507 +0000 UTC m=+2076.347880495" watchObservedRunningTime="2026-10-30 20:54:10.202054241 +0000 UTC m=+2076.351105227"
Oct 30 20:54:11.537412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:11.537366    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:11.929478 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:11.929355    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:16.850317 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:16.850284    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:19.474990 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:19.474958    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:19.668157 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:19.668132    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:20.086158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:20.086128    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:54:21.092513 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:21.092486    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:54:21.600222 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:21.600182    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:25.590718 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.590685    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"]
Oct 30 20:54:25.591146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.590888    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2" podUID="8a024071-e692-4a4e-9fc8-65a4898c7c09" containerName="maas-api" containerID="cri-o://1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345" gracePeriod=30
Oct 30 20:54:25.713756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.713717    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:25.879094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.879067    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:54:25.972563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.967379    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-metrics-certs\") pod \"8a024071-e692-4a4e-9fc8-65a4898c7c09\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") "
Oct 30 20:54:25.972563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.967512    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a024071-e692-4a4e-9fc8-65a4898c7c09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a024071-e692-4a4e-9fc8-65a4898c7c09-openshift-service-ca\") pod \"8a024071-e692-4a4e-9fc8-65a4898c7c09\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") "
Oct 30 20:54:25.972563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.967600    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a024071-e692-4a4e-9fc8-65a4898c7c09-kube-api-access-lbhgq\" (UniqueName: \"kubernetes.io/projected/8a024071-e692-4a4e-9fc8-65a4898c7c09-kube-api-access-lbhgq\") pod \"8a024071-e692-4a4e-9fc8-65a4898c7c09\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") "
Oct 30 20:54:25.972563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.967760    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls\") pod \"8a024071-e692-4a4e-9fc8-65a4898c7c09\" (UID: \"8a024071-e692-4a4e-9fc8-65a4898c7c09\") "
Oct 30 20:54:25.993740 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.993704    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a024071-e692-4a4e-9fc8-65a4898c7c09-openshift-service-ca" pod "8a024071-e692-4a4e-9fc8-65a4898c7c09" (UID: "8a024071-e692-4a4e-9fc8-65a4898c7c09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:25.995162 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:25.995130    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls" pod "8a024071-e692-4a4e-9fc8-65a4898c7c09" (UID: "8a024071-e692-4a4e-9fc8-65a4898c7c09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:26.005805 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.005762    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a024071-e692-4a4e-9fc8-65a4898c7c09-kube-api-access-lbhgq" pod "8a024071-e692-4a4e-9fc8-65a4898c7c09" (UID: "8a024071-e692-4a4e-9fc8-65a4898c7c09"). InnerVolumeSpecName "kube-api-access-lbhgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:26.005940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.005826    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-metrics-certs" pod "8a024071-e692-4a4e-9fc8-65a4898c7c09" (UID: "8a024071-e692-4a4e-9fc8-65a4898c7c09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:26.069970 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.069941    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:26.069970 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.069967    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a024071-e692-4a4e-9fc8-65a4898c7c09-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:26.069970 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.069976    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a024071-e692-4a4e-9fc8-65a4898c7c09-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:26.070171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.069986    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbhgq\" (UniqueName: \"kubernetes.io/projected/8a024071-e692-4a4e-9fc8-65a4898c7c09-kube-api-access-lbhgq\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:26.198641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.198514    5002 generic.go:356] "Generic (PLEG): container finished" podID="8a024071-e692-4a4e-9fc8-65a4898c7c09" containerID="1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345" exitCode=0
Oct 30 20:54:26.198641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.198566    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"
Oct 30 20:54:26.198641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.198565    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2" event={"ID":"8a024071-e692-4a4e-9fc8-65a4898c7c09","Type":"ContainerDied","Data":"1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345"}
Oct 30 20:54:26.198641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.198620    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2" event={"ID":"8a024071-e692-4a4e-9fc8-65a4898c7c09","Type":"ContainerDied","Data":"b09e3f6ef86d14f9b86bfab98b602b085eed7abbe47538e7570d35c69830661e"}
Oct 30 20:54:26.198641 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.198640    5002 scope.go:122] "RemoveContainer" containerID="1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345"
Oct 30 20:54:26.209036 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.209006    5002 scope.go:122] "RemoveContainer" containerID="1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345"
Oct 30 20:54:26.209299 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:54:26.209282    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345\": container with ID starting with 1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345 not found: ID does not exist" containerID="1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345"
Oct 30 20:54:26.209353 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.209308    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345"} err="failed to get container status \"1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345\": rpc error: code = NotFound desc = could not find container \"1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345\": container with ID starting with 1b0451f63c233b6d8b14e91e4fa4ba80a21f6dab144c3dbe3683384e13df5345 not found: ID does not exist"
Oct 30 20:54:26.365444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.365412    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"]
Oct 30 20:54:26.415906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:26.415879    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3d13a-56b75fb96d-wgzb2"]
Oct 30 20:54:27.246380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:27.246348    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:27.580954 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:27.580879    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:27.968252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:27.968215    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a024071-e692-4a4e-9fc8-65a4898c7c09" path="/var/lib/kubelet/pods/8a024071-e692-4a4e-9fc8-65a4898c7c09/volumes"
Oct 30 20:54:28.169683 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:28.169649    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:32.466418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:32.466383    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:34.298603 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:34.298573    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:54:34.308694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:34.308673    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:54:34.540134 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:34.540108    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:34.735470 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:34.735440    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:34.974184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:34.974150    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:35.944815 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:35.944780    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:37.095345 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.095313    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:37.117302 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:54:37.117268    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:54:37.353084 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.353001    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:37.575604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.575572    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"]
Oct 30 20:54:37.576391 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.576369    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8a024071-e692-4a4e-9fc8-65a4898c7c09" containerName="maas-api"
Oct 30 20:54:37.576449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.576394    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a024071-e692-4a4e-9fc8-65a4898c7c09" containerName="maas-api"
Oct 30 20:54:37.576588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.576576    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a024071-e692-4a4e-9fc8-65a4898c7c09" containerName="maas-api"
Oct 30 20:54:37.582300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.582280    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.635696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.635589    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bcef0e-openshift-default-dockercfg-kfg29\""
Oct 30 20:54:37.690193 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.690160    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"]
Oct 30 20:54:37.702856 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.702824    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-envoy\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.702939    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cvz\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-kube-api-access-88cvz\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.702989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89051f8-4d46-4f57-9fb0-66535ddfd473-istiod-ca-cert\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.703030    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-podinfo\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.703081    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-certs\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.703122    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-data\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703228 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.703162    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-token\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.703262    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-socket\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.703327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.703308    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-credential-socket\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.743556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.741130    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:37.804981 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.804943    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cvz\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-kube-api-access-88cvz\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806254 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806210    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89051f8-4d46-4f57-9fb0-66535ddfd473-istiod-ca-cert\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806378 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806288    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-podinfo\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806378 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806355    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-certs\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806409    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-data\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806469    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-token\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806596    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-socket\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806657    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-credential-socket\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.806756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.806711    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-envoy\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.807555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.807514    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-data\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.808907 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.808014    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-socket\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.809065 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.808027    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-credential-socket\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.809196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.808555    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-certs\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.809313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.808815    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89051f8-4d46-4f57-9fb0-66535ddfd473-istiod-ca-cert\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.822182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.820917    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-envoy\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.823957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.823781    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-podinfo\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.868999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.868958    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cvz\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-kube-api-access-88cvz\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.869955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.869928    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-token\") pod \"e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") " pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.896298 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.896228    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:37.951881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:37.951850    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:38.341022 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:38.340990    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"]
Oct 30 20:54:38.342641 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:54:38.342610    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89051f8_4d46_4f57_9fb0_66535ddfd473.slice/crio-7c9115aad091872b887bd982b218dee3b86b5a955a4c4cd9bef2c5fed655e88f WatchSource:0}: Error finding container 7c9115aad091872b887bd982b218dee3b86b5a955a4c4cd9bef2c5fed655e88f: Status 404 returned error can't find the container with id 7c9115aad091872b887bd982b218dee3b86b5a955a4c4cd9bef2c5fed655e88f
Oct 30 20:54:38.345658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:38.345624    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:38.345761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:38.345691    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:38.345761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:38.345719    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:39.296041 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:39.296006    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" event={"ID":"d89051f8-4d46-4f57-9fb0-66535ddfd473","Type":"ContainerStarted","Data":"204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702"}
Oct 30 20:54:39.296226 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:39.296094    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" event={"ID":"d89051f8-4d46-4f57-9fb0-66535ddfd473","Type":"ContainerStarted","Data":"7c9115aad091872b887bd982b218dee3b86b5a955a4c4cd9bef2c5fed655e88f"}
Oct 30 20:54:39.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:39.422048    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podStartSLOduration=2.42202959 podStartE2EDuration="2.42202959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:39.417481407 +0000 UTC m=+2105.566532393" watchObservedRunningTime="2026-10-30 20:54:39.42202959 +0000 UTC m=+2105.571080578"
Oct 30 20:54:39.896454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:39.896409    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:39.902020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:39.901993    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:40.302674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:40.302645    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:40.303684 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:40.303668    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:54:41.393260 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.393225    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"]
Oct 30 20:54:41.393668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.393427    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7" podUID="e311c46e-6d58-4217-96f1-f5d465fcc3a4" containerName="maas-api" containerID="cri-o://01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2" gracePeriod=30
Oct 30 20:54:41.537273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.537241    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:41.776075 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.776041    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:54:41.873848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.873260    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls\") pod \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") "
Oct 30 20:54:41.873848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.873357    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e311c46e-6d58-4217-96f1-f5d465fcc3a4-kube-api-access-lsp6x\" (UniqueName: \"kubernetes.io/projected/e311c46e-6d58-4217-96f1-f5d465fcc3a4-kube-api-access-lsp6x\") pod \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") "
Oct 30 20:54:41.873848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.873398    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-metrics-certs\") pod \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") "
Oct 30 20:54:41.873848 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.873555    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e311c46e-6d58-4217-96f1-f5d465fcc3a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311c46e-6d58-4217-96f1-f5d465fcc3a4-openshift-service-ca\") pod \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\" (UID: \"e311c46e-6d58-4217-96f1-f5d465fcc3a4\") "
Oct 30 20:54:41.874684 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.874350    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e311c46e-6d58-4217-96f1-f5d465fcc3a4-openshift-service-ca" pod "e311c46e-6d58-4217-96f1-f5d465fcc3a4" (UID: "e311c46e-6d58-4217-96f1-f5d465fcc3a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:41.875495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.875470    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311c46e-6d58-4217-96f1-f5d465fcc3a4-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:41.880207 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.880165    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls" pod "e311c46e-6d58-4217-96f1-f5d465fcc3a4" (UID: "e311c46e-6d58-4217-96f1-f5d465fcc3a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:41.893036 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.892990    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e311c46e-6d58-4217-96f1-f5d465fcc3a4-kube-api-access-lsp6x" pod "e311c46e-6d58-4217-96f1-f5d465fcc3a4" (UID: "e311c46e-6d58-4217-96f1-f5d465fcc3a4"). InnerVolumeSpecName "kube-api-access-lsp6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:41.902554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.896709    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-metrics-certs" pod "e311c46e-6d58-4217-96f1-f5d465fcc3a4" (UID: "e311c46e-6d58-4217-96f1-f5d465fcc3a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:41.925172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.925146    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:54:41.978057 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.978033    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:41.978057 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.978057    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsp6x\" (UniqueName: \"kubernetes.io/projected/e311c46e-6d58-4217-96f1-f5d465fcc3a4-kube-api-access-lsp6x\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:41.978203 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:41.978068    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e311c46e-6d58-4217-96f1-f5d465fcc3a4-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:42.239092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.239057    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:42.317351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.317315    5002 generic.go:356] "Generic (PLEG): container finished" podID="e311c46e-6d58-4217-96f1-f5d465fcc3a4" containerID="01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2" exitCode=0
Oct 30 20:54:42.319170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.317805    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"
Oct 30 20:54:42.319170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.318196    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7" event={"ID":"e311c46e-6d58-4217-96f1-f5d465fcc3a4","Type":"ContainerDied","Data":"01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2"}
Oct 30 20:54:42.319170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.318243    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7" event={"ID":"e311c46e-6d58-4217-96f1-f5d465fcc3a4","Type":"ContainerDied","Data":"0bbd0e1bb83c25401b3f32024b0ed883a894ef1daa35c7d1198c2581b7124bc8"}
Oct 30 20:54:42.319170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.318269    5002 scope.go:122] "RemoveContainer" containerID="01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2"
Oct 30 20:54:42.333275 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.333246    5002 scope.go:122] "RemoveContainer" containerID="01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2"
Oct 30 20:54:42.333603 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:54:42.333578    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2\": container with ID starting with 01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2 not found: ID does not exist" containerID="01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2"
Oct 30 20:54:42.333665 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.333615    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2"} err="failed to get container status \"01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2\": rpc error: code = NotFound desc = could not find container \"01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2\": container with ID starting with 01635da6e0dd33b036c28cb55f05d5fdeeb441bf0ed1004fc337d1547c782cc2 not found: ID does not exist"
Oct 30 20:54:42.475355 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.475281    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:42.495906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.495843    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"]
Oct 30 20:54:42.543019 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.542990    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-034eba-798d867c44-254v7"]
Oct 30 20:54:42.686673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.686640    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:42.908552 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:42.908464    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:43.274464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:43.274430    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:43.581114 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:43.579276    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:43.967324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:43.967282    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e311c46e-6d58-4217-96f1-f5d465fcc3a4" path="/var/lib/kubelet/pods/e311c46e-6d58-4217-96f1-f5d465fcc3a4/volumes"
Oct 30 20:54:44.309503 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:44.309439    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:44.806407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:44.806340    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:45.513748 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:45.513717    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:45.774345 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:45.774245    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:45.996901 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:45.996861    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:46.200154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.200068    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:46.265762 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.265735    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"]
Oct 30 20:54:46.266359 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.266347    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e311c46e-6d58-4217-96f1-f5d465fcc3a4" containerName="maas-api"
Oct 30 20:54:46.266416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.266361    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e311c46e-6d58-4217-96f1-f5d465fcc3a4" containerName="maas-api"
Oct 30 20:54:46.266475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.266466    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e311c46e-6d58-4217-96f1-f5d465fcc3a4" containerName="maas-api"
Oct 30 20:54:46.271322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.271304    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.328921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.328901    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"]
Oct 30 20:54:46.347019 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.346989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstgs\" (UniqueName: \"kubernetes.io/projected/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-kube-api-access-rstgs\") pod \"payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.347177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.347157    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.374665 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.374641    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bcef0e\""
Oct 30 20:54:46.422286 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.422254    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:46.445402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.445365    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"]
Oct 30 20:54:46.449265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.449238    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstgs\" (UniqueName: \"kubernetes.io/projected/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-kube-api-access-rstgs\") pod \"payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.449428 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.449327    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.450646 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.450589    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.451013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.450994    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.491376 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.491345    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"]
Oct 30 20:54:46.504727 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.504703    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bcef0e-dockercfg-s4kls\""
Oct 30 20:54:46.507964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.507878    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstgs\" (UniqueName: \"kubernetes.io/projected/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-kube-api-access-rstgs\") pod \"payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.550844 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.550815    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900bc8d-4020-4ec0-8dbe-1dc23712633b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.550987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.550882    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld755\" (UniqueName: \"kubernetes.io/projected/d900bc8d-4020-4ec0-8dbe-1dc23712633b-kube-api-access-ld755\") pod \"payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.588280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.588256    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:46.653671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.652052    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900bc8d-4020-4ec0-8dbe-1dc23712633b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.653671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.652186    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld755\" (UniqueName: \"kubernetes.io/projected/d900bc8d-4020-4ec0-8dbe-1dc23712633b-kube-api-access-ld755\") pod \"payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.653671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.653278    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900bc8d-4020-4ec0-8dbe-1dc23712633b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.711732 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.711703    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld755\" (UniqueName: \"kubernetes.io/projected/d900bc8d-4020-4ec0-8dbe-1dc23712633b-kube-api-access-ld755\") pod \"payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.769908 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.769883    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:46.972226 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:46.972198    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"]
Oct 30 20:54:46.972523 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:54:46.972501    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8812bf72_cfe9_4e13_9196_96e2ab52b2fd.slice/crio-36725ad2f1c47edf72340e9510686dec404f1dcdd885acfc51663f4be72b5c99 WatchSource:0}: Error finding container 36725ad2f1c47edf72340e9510686dec404f1dcdd885acfc51663f4be72b5c99: Status 404 returned error can't find the container with id 36725ad2f1c47edf72340e9510686dec404f1dcdd885acfc51663f4be72b5c99
Oct 30 20:54:47.160254 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:54:47.160225    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd900bc8d_4020_4ec0_8dbe_1dc23712633b.slice/crio-ebccd78423cff98cddaf7b77f6f1d0fb0419e75b40dbb8b53b7da84ef7aa1100 WatchSource:0}: Error finding container ebccd78423cff98cddaf7b77f6f1d0fb0419e75b40dbb8b53b7da84ef7aa1100: Status 404 returned error can't find the container with id ebccd78423cff98cddaf7b77f6f1d0fb0419e75b40dbb8b53b7da84ef7aa1100
Oct 30 20:54:47.160887 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.160865    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"]
Oct 30 20:54:47.363402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.363366    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q" event={"ID":"d900bc8d-4020-4ec0-8dbe-1dc23712633b","Type":"ContainerStarted","Data":"b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793"}
Oct 30 20:54:47.363580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.363431    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q" event={"ID":"d900bc8d-4020-4ec0-8dbe-1dc23712633b","Type":"ContainerStarted","Data":"ebccd78423cff98cddaf7b77f6f1d0fb0419e75b40dbb8b53b7da84ef7aa1100"}
Oct 30 20:54:47.363580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.363462    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:47.365082 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.365057    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" event={"ID":"8812bf72-cfe9-4e13-9196-96e2ab52b2fd","Type":"ContainerStarted","Data":"c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7"}
Oct 30 20:54:47.365187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.365099    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" event={"ID":"8812bf72-cfe9-4e13-9196-96e2ab52b2fd","Type":"ContainerStarted","Data":"36725ad2f1c47edf72340e9510686dec404f1dcdd885acfc51663f4be72b5c99"}
Oct 30 20:54:47.365187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.365119    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:47.483181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.483097    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q" podStartSLOduration=1.4830846549999999 podStartE2EDuration="1.483084655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:47.479736908 +0000 UTC m=+2113.628787896" watchObservedRunningTime="2026-10-30 20:54:47.483084655 +0000 UTC m=+2113.632135642"
Oct 30 20:54:47.604830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:47.604752    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" podStartSLOduration=1.6047320699999998 podStartE2EDuration="1.60473207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:47.597155635 +0000 UTC m=+2113.746206629" watchObservedRunningTime="2026-10-30 20:54:47.60473207 +0000 UTC m=+2113.753783058"
Oct 30 20:54:50.117079 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:50.117036    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:50.315273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:50.315241    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:50.630812 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:50.630782    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:51.810881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:51.810841    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:52.764585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:52.764525    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:53.401440 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.401396    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"]
Oct 30 20:54:53.406822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.406801    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.479716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.479684    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"]
Oct 30 20:54:53.510819 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.510790    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-65948c-openshift-default-dockercfg-76rpr\""
Oct 30 20:54:53.532934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.532899    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:53.559112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559064    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-certs\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559118    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-credential-socket\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559146    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-podinfo\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559198    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-data\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559240    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-envoy\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559313 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559289    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-token\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559637 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559399    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-socket\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559637 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559495    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkk4\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-kube-api-access-bfkk4\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.559637 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.559581    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istiod-ca-cert\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.664716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.664638    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-socket\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.664899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.664731    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkk4\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-kube-api-access-bfkk4\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.664899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.664779    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istiod-ca-cert\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.664899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.664815    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-certs\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.664899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.664834    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-credential-socket\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.664899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.664852    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-podinfo\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.664899 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.664901    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-data\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.665235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.665048    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-socket\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.665235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.665045    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-envoy\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.665235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.665140    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-token\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.665235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.665177    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-certs\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.665235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.665211    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-credential-socket\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.665501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.665306    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-data\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.665586 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.665552    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istiod-ca-cert\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.667305 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.667283    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-envoy\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.667581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.667557    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-podinfo\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.724333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.724307    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkk4\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-kube-api-access-bfkk4\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.724571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.724525    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-token\") pod \"e2e-api-b-65948c-openshift-default-b44f97585-gz54k\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") " pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:53.790557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:53.790438    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:54.022262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.022227    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:54.081634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.081576    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:54.313517 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.313423    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:54.483327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.483285    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"]
Oct 30 20:54:54.509634 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:54:54.508962    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88071f20_7d08_4bda_a6c6_65eccb3ab8e7.slice/crio-e747f228265b4bdefa4c512e6ddd8ba01de635b37d9032007c177f0a93225ee4 WatchSource:0}: Error finding container e747f228265b4bdefa4c512e6ddd8ba01de635b37d9032007c177f0a93225ee4: Status 404 returned error can't find the container with id e747f228265b4bdefa4c512e6ddd8ba01de635b37d9032007c177f0a93225ee4
Oct 30 20:54:54.510103 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.509858    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:54.513772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.512676    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:54.513772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.512763    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:54.513772 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:54.512831    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:55.425153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:55.425114    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" event={"ID":"88071f20-7d08-4bda-a6c6-65eccb3ab8e7","Type":"ContainerStarted","Data":"d85cb981d53fc8363179c89b115c1acb69403a8c638a28a090bd0b0de71891f2"}
Oct 30 20:54:55.425347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:55.425172    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" event={"ID":"88071f20-7d08-4bda-a6c6-65eccb3ab8e7","Type":"ContainerStarted","Data":"e747f228265b4bdefa4c512e6ddd8ba01de635b37d9032007c177f0a93225ee4"}
Oct 30 20:54:55.458570 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:55.444997    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:55.652067 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:55.650364    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podStartSLOduration=2.650344699 podStartE2EDuration="2.650344699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:55.597948534 +0000 UTC m=+2121.746999522" watchObservedRunningTime="2026-10-30 20:54:55.650344699 +0000 UTC m=+2121.799395686"
Oct 30 20:54:56.023241 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:56.023211    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:56.029398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:56.029369    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:56.433961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:56.433876    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:56.435014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:56.434982    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:54:58.374522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:58.374493    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 20:54:58.374939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:58.374607    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 20:54:58.422894 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:58.422868    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:58.805145 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:58.805109    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:59.007807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:54:59.007773    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:00.673909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:00.673329    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:01.768679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:01.768646    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:01.955553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:01.955511    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:02.325556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.324614    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"]
Oct 30 20:55:02.330325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.330271    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.359153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.359113    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"]
Oct 30 20:55:02.363457 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.363421    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99x5\" (UniqueName: \"kubernetes.io/projected/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-kube-api-access-f99x5\") pod \"payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") " pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.363629 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.363592    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") " pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.403808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.403763    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-65948c\""
Oct 30 20:55:02.403972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.403899    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-65948c-dockercfg-z7cvm\""
Oct 30 20:55:02.464890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.464853    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") " pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.465080 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.465016    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99x5\" (UniqueName: \"kubernetes.io/projected/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-kube-api-access-f99x5\") pod \"payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") " pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.465609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.465589    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") " pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.488184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.488150    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"]
Oct 30 20:55:02.492487 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.492463    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:02.524060 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.524036    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"]
Oct 30 20:55:02.524817 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.524770    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99x5\" (UniqueName: \"kubernetes.io/projected/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-kube-api-access-f99x5\") pod \"payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") " pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.565883 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.565854    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:02.566043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.566026    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5pft\" (UniqueName: \"kubernetes.io/projected/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-kube-api-access-l5pft\") pod \"payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:02.660023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.659946    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:02.667580 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.667552    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5pft\" (UniqueName: \"kubernetes.io/projected/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-kube-api-access-l5pft\") pod \"payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:02.667720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.667672    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:02.668374 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.668348    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:02.729010 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.726852    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5pft\" (UniqueName: \"kubernetes.io/projected/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-kube-api-access-l5pft\") pod \"payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:02.807733 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:02.807699    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:03.046177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.046147    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"]
Oct 30 20:55:03.047851 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:55:03.047821    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6cbd36_468d_4fa2_ad6d_279a319bc7ba.slice/crio-777640ce44335f55a6a7b15a6ba9247c24cf6e55239ba9df6e0806d278425424 WatchSource:0}: Error finding container 777640ce44335f55a6a7b15a6ba9247c24cf6e55239ba9df6e0806d278425424: Status 404 returned error can't find the container with id 777640ce44335f55a6a7b15a6ba9247c24cf6e55239ba9df6e0806d278425424
Oct 30 20:55:03.186179 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:55:03.186146    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d69e2c_d227_4e0c_b4f0_8ee8e8c72bda.slice/crio-7b0da0f484fe08e201e1b36f6fa0687100dfa281f470aae7e18fb3d996e0a373 WatchSource:0}: Error finding container 7b0da0f484fe08e201e1b36f6fa0687100dfa281f470aae7e18fb3d996e0a373: Status 404 returned error can't find the container with id 7b0da0f484fe08e201e1b36f6fa0687100dfa281f470aae7e18fb3d996e0a373
Oct 30 20:55:03.186400 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.186379    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"]
Oct 30 20:55:03.335947 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.335891    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:03.490544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.490495    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l" event={"ID":"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda","Type":"ContainerStarted","Data":"0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14"}
Oct 30 20:55:03.490719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.490567    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l" event={"ID":"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda","Type":"ContainerStarted","Data":"7b0da0f484fe08e201e1b36f6fa0687100dfa281f470aae7e18fb3d996e0a373"}
Oct 30 20:55:03.490719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.490592    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:03.492250 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.492223    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw" event={"ID":"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba","Type":"ContainerStarted","Data":"a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95"}
Oct 30 20:55:03.492356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.492268    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw" event={"ID":"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba","Type":"ContainerStarted","Data":"777640ce44335f55a6a7b15a6ba9247c24cf6e55239ba9df6e0806d278425424"}
Oct 30 20:55:03.492356 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.492288    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:03.524572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.524502    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:03.612417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.612335    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l" podStartSLOduration=1.612319125 podStartE2EDuration="1.612319125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:03.610879613 +0000 UTC m=+2129.759930605" watchObservedRunningTime="2026-10-30 20:55:03.612319125 +0000 UTC m=+2129.761370112"
Oct 30 20:55:03.737328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:03.737266    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw" podStartSLOduration=1.737247216 podStartE2EDuration="1.737247216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:03.73214166 +0000 UTC m=+2129.881192665" watchObservedRunningTime="2026-10-30 20:55:03.737247216 +0000 UTC m=+2129.886298205"
Oct 30 20:55:05.101724 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.101691    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"]
Oct 30 20:55:05.102153 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.101890    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" podUID="b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" containerName="maas-api" containerID="cri-o://bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123" gracePeriod=30
Oct 30 20:55:05.244988 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.244956    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:05.248560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.248518    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" podUID="b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.124:8443/health\": dial tcp 10.132.0.124:8443: connect: connection refused"
Oct 30 20:55:05.396004 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.395980    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:55:05.505943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.505911    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-kube-api-access-979r2\" (UniqueName: \"kubernetes.io/projected/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-kube-api-access-979r2\") pod \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") "
Oct 30 20:55:05.506128 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.506045    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-openshift-service-ca\") pod \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") "
Oct 30 20:55:05.506128 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.506095    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls\") pod \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") "
Oct 30 20:55:05.506128 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.506124    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-metrics-certs\") pod \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\" (UID: \"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd\") "
Oct 30 20:55:05.506450 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.506421    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-openshift-service-ca" pod "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" (UID: "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:05.506746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.506699    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:05.510324 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.510258    5002 generic.go:356] "Generic (PLEG): container finished" podID="b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" containerID="bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123" exitCode=0
Oct 30 20:55:05.510460 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.510360    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"
Oct 30 20:55:05.510813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.510595    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" event={"ID":"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd","Type":"ContainerDied","Data":"bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123"}
Oct 30 20:55:05.510813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.510654    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc" event={"ID":"b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd","Type":"ContainerDied","Data":"32890b99dea8d245832045adc1b1657c8477388c468d59c9a18ca794ac731497"}
Oct 30 20:55:05.510813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.510685    5002 scope.go:122] "RemoveContainer" containerID="bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123"
Oct 30 20:55:05.558599 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.537093    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-kube-api-access-979r2" pod "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" (UID: "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd"). InnerVolumeSpecName "kube-api-access-979r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:05.558599 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.541667    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls" pod "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" (UID: "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:05.558599 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.545337    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-metrics-certs" pod "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" (UID: "b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:05.559874 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.559070    5002 scope.go:122] "RemoveContainer" containerID="bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123"
Oct 30 20:55:05.559874 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:55:05.559524    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123\": container with ID starting with bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123 not found: ID does not exist" containerID="bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123"
Oct 30 20:55:05.559874 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.559586    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123"} err="failed to get container status \"bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123\": rpc error: code = NotFound desc = could not find container \"bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123\": container with ID starting with bd9ebbe615649b62370c847a0319997d0d6b344b15c407ca731cb826153bb123 not found: ID does not exist"
Oct 30 20:55:05.608068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.608039    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979r2\" (UniqueName: \"kubernetes.io/projected/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-kube-api-access-979r2\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:05.608068 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.608067    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:05.608257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.608077    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:05.936285 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.936248    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"]
Oct 30 20:55:05.939647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.939622    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab9c38-7b6f949f88-mkhgc"]
Oct 30 20:55:05.978219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:05.978188    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" path="/var/lib/kubelet/pods/b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd/volumes"
Oct 30 20:55:06.758499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.758466    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"]
Oct 30 20:55:06.759142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.759125    5002 cpu_manager.go:414] "containerMap: removing container" podUID="b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" containerName="maas-api"
Oct 30 20:55:06.759190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.759144    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" containerName="maas-api"
Oct 30 20:55:06.759243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.759235    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98af6c7-b0ba-42f5-9a70-16d7a4ee03dd" containerName="maas-api"
Oct 30 20:55:06.764085 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.764070    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.821372 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.821347    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"]
Oct 30 20:55:06.828127 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.828099    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772b740d-1836-4496-b4b9-9ff6f1a58b4d-openshift-service-ca\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.828246 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.828173    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kpj9\" (UniqueName: \"kubernetes.io/projected/772b740d-1836-4496-b4b9-9ff6f1a58b4d-kube-api-access-4kpj9\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.828308 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.828256    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-metrics-certs\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.828422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.828405    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.867142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.867124    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-27219c73-serving-cert\""
Oct 30 20:55:06.879537 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.879510    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:06.929972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.929945    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772b740d-1836-4496-b4b9-9ff6f1a58b4d-openshift-service-ca\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.930124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.930011    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kpj9\" (UniqueName: \"kubernetes.io/projected/772b740d-1836-4496-b4b9-9ff6f1a58b4d-kube-api-access-4kpj9\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.930124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.930050    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-metrics-certs\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.930479 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.930457    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.930609 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:55:06.930587    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-serving-cert: secret "maas-api-e2e-mt-27219c73-serving-cert" not found
Oct 30 20:55:06.930716 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:55:06.930645    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls podName:772b740d-1836-4496-b4b9-9ff6f1a58b4d nodeName:}" failed. No retries permitted until 2026-10-30 20:55:07.430627692 +0000 UTC m=+2133.579678659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls") pod "maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68" (UID: "772b740d-1836-4496-b4b9-9ff6f1a58b4d") : secret "maas-api-e2e-mt-27219c73-serving-cert" not found
Oct 30 20:55:06.930808 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.930778    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772b740d-1836-4496-b4b9-9ff6f1a58b4d-openshift-service-ca\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.933304 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.933287    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-metrics-certs\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:06.988964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:06.988936    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kpj9\" (UniqueName: \"kubernetes.io/projected/772b740d-1836-4496-b4b9-9ff6f1a58b4d-kube-api-access-4kpj9\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:07.096495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:07.096402    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:07.121574 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:55:07.121510    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:55:07.284054 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:07.284016    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:07.437171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:07.437093    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:07.439640 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:07.439621    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls\") pod \"maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:07.466417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:07.466389    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:07.678802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:07.678767    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:07.904801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:07.904770    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:08.062716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:08.062687    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"]
Oct 30 20:55:08.062863 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:55:08.062820    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772b740d_1836_4496_b4b9_9ff6f1a58b4d.slice/crio-95d65d85662de56af26cd3f8dfc25a030704deecb49e0859a64b6d36a5c259e1 WatchSource:0}: Error finding container 95d65d85662de56af26cd3f8dfc25a030704deecb49e0859a64b6d36a5c259e1: Status 404 returned error can't find the container with id 95d65d85662de56af26cd3f8dfc25a030704deecb49e0859a64b6d36a5c259e1
Oct 30 20:55:08.533661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:08.533630    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68" event={"ID":"772b740d-1836-4496-b4b9-9ff6f1a58b4d","Type":"ContainerStarted","Data":"95d65d85662de56af26cd3f8dfc25a030704deecb49e0859a64b6d36a5c259e1"}
Oct 30 20:55:09.542061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:09.542019    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68" event={"ID":"772b740d-1836-4496-b4b9-9ff6f1a58b4d","Type":"ContainerStarted","Data":"481e2294bd309b10be3e37da85a5affd438e801906465d8f0fa73e1bdc967372"}
Oct 30 20:55:09.542492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:09.542110    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:09.661291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:09.661245    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68" podStartSLOduration=2.832189949 podStartE2EDuration="3.661231715s" totalImagesPullingTime="829.041766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:09.660837174 +0000 UTC m=+2135.809888161" watchObservedRunningTime="2026-10-30 20:55:09.661231715 +0000 UTC m=+2135.810282701"
Oct 30 20:55:10.339341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:10.339303    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"]
Oct 30 20:55:10.689271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:10.689162    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:10.874473 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:10.874438    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:11.730676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:11.730649    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:13.309829 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:13.309795    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:13.567393 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:13.567316    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:13.747568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:13.747503    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:14.267697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:14.267665    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:14.503380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:14.503355    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:55:14.503840 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:14.503417    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:55:14.509341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:14.509317    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:14.707124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:14.707036    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:14.930627 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:14.929503    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:15.340059 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.340021    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq" podUID="5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" containerName="payload-pre-processing" containerID="cri-o://99615a8d876011af3710c7557fe34fb4154cab13ca1737a28df4aaae64bda276" gracePeriod=25
Oct 30 20:55:15.556235 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.556200    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:55:15.612419 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.612377    5002 generic.go:356] "Generic (PLEG): container finished" podID="5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" containerID="99615a8d876011af3710c7557fe34fb4154cab13ca1737a28df4aaae64bda276" exitCode=0
Oct 30 20:55:15.612594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.612443    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq" event={"ID":"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77","Type":"ContainerDied","Data":"99615a8d876011af3710c7557fe34fb4154cab13ca1737a28df4aaae64bda276"}
Oct 30 20:55:15.612594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.612496    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq" event={"ID":"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77","Type":"ContainerDied","Data":"af7f89e372b75dcde68540643f3c288c21556481ee405fc599b54ef4be327eb7"}
Oct 30 20:55:15.612594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.612522    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7f89e372b75dcde68540643f3c288c21556481ee405fc599b54ef4be327eb7"
Oct 30 20:55:15.625184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.625153    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:55:15.744266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.744231    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-plugins-config-volume\") pod \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") "
Oct 30 20:55:15.744448 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.744326    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-kube-api-access-t9smq\" (UniqueName: \"kubernetes.io/projected/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-kube-api-access-t9smq\") pod \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\" (UID: \"5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77\") "
Oct 30 20:55:15.744514 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.744449    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-plugins-config-volume" pod "5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" (UID: "5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:15.744987 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.744963    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:15.746877 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.746845    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-kube-api-access-t9smq" pod "5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" (UID: "5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77"). InnerVolumeSpecName "kube-api-access-t9smq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:15.847763 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.847676    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9smq\" (UniqueName: \"kubernetes.io/projected/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77-kube-api-access-t9smq\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:15.854551 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:15.854509    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:16.057581 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:16.057548    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:16.270190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:16.270160    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:16.595179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:16.595107    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:16.619817 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:16.619789    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"
Oct 30 20:55:16.794837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:16.794799    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"]
Oct 30 20:55:16.846503 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:16.846430    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq"]
Oct 30 20:55:17.966929 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:17.966895    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" path="/var/lib/kubelet/pods/5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77/volumes"
Oct 30 20:55:18.701921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:18.701887    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:20.380492 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:20.380449    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:20.664915 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:20.664827    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:20.904608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:20.904495    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:21.684573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:21.684480    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:22.275172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:22.275134    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:22.512700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:22.512663    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:22.927862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:22.927831    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:23.500553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:23.500484    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:25.598769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:25.598726    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:55:26.746279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:26.746233    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:27.047450 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:27.047366    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:27.253782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:27.253755    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:27.832573 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:27.832513    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:29.716134 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:29.716105    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:29.890590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:29.890558    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:30.300499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:30.300461    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:30.485050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:30.484998    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:30.671774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:30.671654    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:30.896163 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:30.896126    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:34.556657 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:34.556623    5002 scope.go:122] "RemoveContainer" containerID="99615a8d876011af3710c7557fe34fb4154cab13ca1737a28df4aaae64bda276"
Oct 30 20:55:35.161926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:35.161893    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:36.249626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:36.249594    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:36.430013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:36.429984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:36.810104 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:36.810050    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:36.996407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:36.996360    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:37.116562 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:55:37.116458    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:55:37.215614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:37.215490    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:40.028333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:40.028301    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:41.320200 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:41.320147    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:41.498863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:41.498830    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:45.294476 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:45.294440    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:48.534992 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:48.534960    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:50.367462 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:50.367424    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:50.543729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:50.543696    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:52.155544 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:52.155404    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"]
Oct 30 20:55:52.549462 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:52.549430    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:52.726879 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:52.726844    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:54.513639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:54.513607    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:55.538078 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:55.538044    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:57.156340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.156303    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh" podUID="4dd208ad-a560-4fe9-b756-a0bd171b70fb" containerName="payload-processing" containerID="cri-o://5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03" gracePeriod=25
Oct 30 20:55:57.433632 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.433609    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:55:57.577274 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.577247    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd208ad-a560-4fe9-b756-a0bd171b70fb-kube-api-access-s7nks\" (UniqueName: \"kubernetes.io/projected/4dd208ad-a560-4fe9-b756-a0bd171b70fb-kube-api-access-s7nks\") pod \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") "
Oct 30 20:55:57.577454 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.577384    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd208ad-a560-4fe9-b756-a0bd171b70fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd208ad-a560-4fe9-b756-a0bd171b70fb-plugins-config-volume\") pod \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\" (UID: \"4dd208ad-a560-4fe9-b756-a0bd171b70fb\") "
Oct 30 20:55:57.578217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.578082    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd208ad-a560-4fe9-b756-a0bd171b70fb-plugins-config-volume" pod "4dd208ad-a560-4fe9-b756-a0bd171b70fb" (UID: "4dd208ad-a560-4fe9-b756-a0bd171b70fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:57.579769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.579730    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd208ad-a560-4fe9-b756-a0bd171b70fb-kube-api-access-s7nks" pod "4dd208ad-a560-4fe9-b756-a0bd171b70fb" (UID: "4dd208ad-a560-4fe9-b756-a0bd171b70fb"). InnerVolumeSpecName "kube-api-access-s7nks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:57.678920 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.678863    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nks\" (UniqueName: \"kubernetes.io/projected/4dd208ad-a560-4fe9-b756-a0bd171b70fb-kube-api-access-s7nks\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:57.678920 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.678889    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd208ad-a560-4fe9-b756-a0bd171b70fb-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:57.946208 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.946118    5002 generic.go:356] "Generic (PLEG): container finished" podID="4dd208ad-a560-4fe9-b756-a0bd171b70fb" containerID="5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03" exitCode=0
Oct 30 20:55:57.946208 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.946165    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"
Oct 30 20:55:57.946451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.946204    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh" event={"ID":"4dd208ad-a560-4fe9-b756-a0bd171b70fb","Type":"ContainerDied","Data":"5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03"}
Oct 30 20:55:57.946451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.946265    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh" event={"ID":"4dd208ad-a560-4fe9-b756-a0bd171b70fb","Type":"ContainerDied","Data":"35d7418d8b7fd732a357ece04a87baa2cf0b621cfb2e62c1b1f0446ce9752610"}
Oct 30 20:55:57.946451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.946296    5002 scope.go:122] "RemoveContainer" containerID="5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03"
Oct 30 20:55:57.957697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.957676    5002 scope.go:122] "RemoveContainer" containerID="5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03"
Oct 30 20:55:57.957973 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:55:57.957954    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03\": container with ID starting with 5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03 not found: ID does not exist" containerID="5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03"
Oct 30 20:55:57.958026 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:57.957981    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03"} err="failed to get container status \"5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03\": rpc error: code = NotFound desc = could not find container \"5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03\": container with ID starting with 5fa114e81fe0c039f8a353608cac4176f1509dc670eef8a768f2348837d52e03 not found: ID does not exist"
Oct 30 20:55:58.118110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:58.118074    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"]
Oct 30 20:55:58.130102 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:58.130070    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh"]
Oct 30 20:55:59.967090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:55:59.967057    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd208ad-a560-4fe9-b756-a0bd171b70fb" path="/var/lib/kubelet/pods/4dd208ad-a560-4fe9-b756-a0bd171b70fb/volumes"
Oct 30 20:56:00.533606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:00.533572    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:05.048807 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:05.048775    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:06.672836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:06.672801    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:06.896801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:06.896766    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:07.092357 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:07.092321    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:07.138068 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:07.138024    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:56:07.276585 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:07.276551    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:07.602169 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:07.602138    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:08.612952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:08.612916    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:56:12.020853 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:12.020817    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:13.707749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:13.707709    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:13.886451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:13.886407    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:14.181985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:14.181948    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:14.996302 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:14.996271    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:15.474802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.474766    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"]
Oct 30 20:56:15.475762 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.475740    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" containerName="payload-pre-processing"
Oct 30 20:56:15.475825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.475766    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" containerName="payload-pre-processing"
Oct 30 20:56:15.475861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.475831    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4dd208ad-a560-4fe9-b756-a0bd171b70fb" containerName="payload-processing"
Oct 30 20:56:15.475861 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.475839    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd208ad-a560-4fe9-b756-a0bd171b70fb" containerName="payload-processing"
Oct 30 20:56:15.476035 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.476023    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd208ad-a560-4fe9-b756-a0bd171b70fb" containerName="payload-processing"
Oct 30 20:56:15.476072 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.476045    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8b35af-bf7f-4ad7-b885-bc3c9cdc3d77" containerName="payload-pre-processing"
Oct 30 20:56:15.481693 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.481670    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.543965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.543872    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"]
Oct 30 20:56:15.582347 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.582317    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cd2d96-openshift-default-dockercfg-s6sqx\""
Oct 30 20:56:15.605735 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.605702    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619207    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-token\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619254    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-credential-socket\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619473    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4d728a-78c0-405c-9907-6df33d6bd997-istiod-ca-cert\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619517    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-certs\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619665    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-podinfo\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619710    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-envoy\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619775    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-socket\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.619976    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-data\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.623602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.620154    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctmt\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-kube-api-access-9ctmt\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721565    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-data\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721629    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctmt\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-kube-api-access-9ctmt\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721718    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-token\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721770    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-credential-socket\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721841    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4d728a-78c0-405c-9907-6df33d6bd997-istiod-ca-cert\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721874    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-certs\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721933    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-podinfo\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.721979    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-envoy\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.722062    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-socket\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.722550    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-socket\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.722799    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-data\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.723567    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-credential-socket\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.724193    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4d728a-78c0-405c-9907-6df33d6bd997-istiod-ca-cert\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.724572 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.724405    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-certs\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.728132 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.728032    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-envoy\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.728852 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.728799    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-podinfo\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.780177 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.780151    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-token\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.780606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.780586    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctmt\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-kube-api-access-9ctmt\") pod \"e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") " pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.795889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.795867    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:15.901960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:15.901928    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:16.258755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:16.257980    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"]
Oct 30 20:56:16.260726 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:16.260687    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:16.260845 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:16.260757    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:16.260845 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:16.260786    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:16.791095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:16.791063    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:17.108782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:17.108706    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" event={"ID":"2a4d728a-78c0-405c-9907-6df33d6bd997","Type":"ContainerStarted","Data":"cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4"}
Oct 30 20:56:17.108782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:17.108761    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" event={"ID":"2a4d728a-78c0-405c-9907-6df33d6bd997","Type":"ContainerStarted","Data":"3324b921a027fef7c332780ba75b6cc9956db6acbed22508122b4b72deb0aa30"}
Oct 30 20:56:17.233906 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:17.233836    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" podStartSLOduration=2.233819377 podStartE2EDuration="2.233819377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:17.231639619 +0000 UTC m=+2203.380690607" watchObservedRunningTime="2026-10-30 20:56:17.233819377 +0000 UTC m=+2203.382870364"
Oct 30 20:56:17.796292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:17.796263    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:17.803395 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:17.803371    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:18.118728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:18.118639    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:18.119674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:18.119653    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:56:19.127823 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:19.127741    5002 generic.go:356] "Generic (PLEG): container finished" podID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerID="204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702" exitCode=0
Oct 30 20:56:19.128226 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:19.127820    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" event={"ID":"d89051f8-4d46-4f57-9fb0-66535ddfd473","Type":"ContainerDied","Data":"204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702"}
Oct 30 20:56:19.128446 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:19.128428    5002 scope.go:122] "RemoveContainer" containerID="204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702"
Oct 30 20:56:19.130103 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:19.130067    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:19.130205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:19.130157    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:19.130262 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:19.130204    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:20.060516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.060470    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"]
Oct 30 20:56:20.060827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.060785    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" containerID="cri-o://03d0ba1750e578350b8971ad09407021e614854bd4a913910be9c2502d77b1a8" gracePeriod=30
Oct 30 20:56:20.117614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.116702    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"]
Oct 30 20:56:20.121622 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.118155    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" containerID="cri-o://fa64e90450ab5c6cad68c33ad20668633fb902bd942814c443e0bf9943eb412d" gracePeriod=30
Oct 30 20:56:20.134553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.125005    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:20.172794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.171317    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"]
Oct 30 20:56:20.172794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.171655    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" containerID="cri-o://533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498" gracePeriod=30
Oct 30 20:56:20.186404 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.183129    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" event={"ID":"d89051f8-4d46-4f57-9fb0-66535ddfd473","Type":"ContainerStarted","Data":"d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97"}
Oct 30 20:56:20.350626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.339263    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:20.896499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.896467    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:56:20.897982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.897956    5002 patch_prober.go:29] interesting pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:56:20.898095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:20.898003    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:56:21.889955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:21.889919    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:21.897160 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:21.897124    5002 patch_prober.go:29] interesting pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:56:21.897374 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:21.897195    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:56:22.896620 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:22.896584    5002 patch_prober.go:29] interesting pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:56:22.897042 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:22.896649    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:56:23.241742 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.241687    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:56:23.242071 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.242018    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:56:23.475824 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.475783    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:56:23.475980 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.475840    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:56:23.538611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.536697    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:56:23.538611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.536761    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:56:23.896713 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.896640    5002 patch_prober.go:29] interesting pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:56:23.897485 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:23.896734    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:56:24.897364 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:24.897333    5002 patch_prober.go:29] interesting pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:56:24.897804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:24.897386    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:56:25.148681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.148605    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:25.243556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.243268    5002 generic.go:356] "Generic (PLEG): container finished" podID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerID="03d0ba1750e578350b8971ad09407021e614854bd4a913910be9c2502d77b1a8" exitCode=0
Oct 30 20:56:25.243556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.243380    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" event={"ID":"ada2cf2a-29a4-4042-b8be-9536203e8703","Type":"ContainerDied","Data":"03d0ba1750e578350b8971ad09407021e614854bd4a913910be9c2502d77b1a8"}
Oct 30 20:56:25.247555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.245507    5002 generic.go:356] "Generic (PLEG): container finished" podID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerID="fa64e90450ab5c6cad68c33ad20668633fb902bd942814c443e0bf9943eb412d" exitCode=0
Oct 30 20:56:25.247555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.245649    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" event={"ID":"de5fefae-1b0e-4774-988c-fbe7be25d757","Type":"ContainerDied","Data":"fa64e90450ab5c6cad68c33ad20668633fb902bd942814c443e0bf9943eb412d"}
Oct 30 20:56:25.321559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.317646    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:56:25.387796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.371519    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:25.445174 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.444584    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:56:25.489778 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.489732    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-podinfo\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.489965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.489820    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-envoy\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.489965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.489866    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-certs\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.489965 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.489913    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-credential-socket\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.490144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.489978    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-kube-api-access-ddqg6\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-kube-api-access-ddqg6\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.490144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.490023    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-token\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-token\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.490144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.490089    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-data\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.490729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.490701    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-credential-socket" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.491464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.491375    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-certs" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.491819 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.491711    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-data" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.492067 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.492043    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-socket" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.492204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.492175    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-socket\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.492281 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.492263    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada2cf2a-29a4-4042-b8be-9536203e8703-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada2cf2a-29a4-4042-b8be-9536203e8703-istiod-ca-cert\") pod \"ada2cf2a-29a4-4042-b8be-9536203e8703\" (UID: \"ada2cf2a-29a4-4042-b8be-9536203e8703\") "
Oct 30 20:56:25.493015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.492991    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.493111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.493020    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.493111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.493035    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.493111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.493048    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.493402 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.493380    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada2cf2a-29a4-4042-b8be-9536203e8703-istiod-ca-cert" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:25.493976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.493955    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-kube-api-access-ddqg6" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "kube-api-access-ddqg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:25.508618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.496980    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-token" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:25.508618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.500817    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-envoy" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.508618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.502202    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-podinfo" pod "ada2cf2a-29a4-4042-b8be-9536203e8703" (UID: "ada2cf2a-29a4-4042-b8be-9536203e8703"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:56:25.590012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.589980    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:56:25.593875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.593850    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.593875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.593874    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.594073 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.593884    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddqg6\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-kube-api-access-ddqg6\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.594073 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.593895    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada2cf2a-29a4-4042-b8be-9536203e8703-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.594073 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.593905    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada2cf2a-29a4-4042-b8be-9536203e8703-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.691277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.691248    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"]
Oct 30 20:56:25.692025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.691996    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy"
Oct 30 20:56:25.692025 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.692018    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy"
Oct 30 20:56:25.692181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.692073    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy"
Oct 30 20:56:25.692181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.692081    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy"
Oct 30 20:56:25.692291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.692249    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" containerName="istio-proxy"
Oct 30 20:56:25.692291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.692270    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" containerName="istio-proxy"
Oct 30 20:56:25.694278 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694255    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-data\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694392 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694311    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9963d70a-6261-48fc-b1b7-aa3138751293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9963d70a-6261-48fc-b1b7-aa3138751293-istio-podinfo\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694404    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-certs\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694453 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694438    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-socket\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694605 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694579    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-data" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.694668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694584    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-kube-api-access-wxqk8\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-kube-api-access-wxqk8\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694713    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-envoy\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694787 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694771    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-credential-socket\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694849 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694801    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-socket" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.694849 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694811    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-certs" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.694961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694858    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-istio-token\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-istio-token\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.694961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.694898    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9963d70a-6261-48fc-b1b7-aa3138751293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9963d70a-6261-48fc-b1b7-aa3138751293-istiod-ca-cert\") pod \"9963d70a-6261-48fc-b1b7-aa3138751293\" (UID: \"9963d70a-6261-48fc-b1b7-aa3138751293\") "
Oct 30 20:56:25.695101 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.695080    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-credential-socket" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.695438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.695405    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9963d70a-6261-48fc-b1b7-aa3138751293-istiod-ca-cert" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:25.695438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.695419    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.695660 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.695438    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.695660 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.695454    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.695660 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.695467    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.697043 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.697022    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-kube-api-access-wxqk8" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "kube-api-access-wxqk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:25.697359 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.697340    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:25.697444 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.697370    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-istio-token" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:25.698366 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.698347    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-envoy" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.698511 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.698490    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9963d70a-6261-48fc-b1b7-aa3138751293-istio-podinfo" pod "9963d70a-6261-48fc-b1b7-aa3138751293" (UID: "9963d70a-6261-48fc-b1b7-aa3138751293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:56:25.763828 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.755095    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"]
Oct 30 20:56:25.768780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.768738    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd2d96-dockercfg-q85jm\""
Oct 30 20:56:25.768950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.768871    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd2d96\""
Oct 30 20:56:25.796781 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.796750    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5js49\" (UniqueName: \"kubernetes.io/projected/8349d596-e27d-46f9-923a-e731adcf5292-kube-api-access-5js49\") pod \"payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:25.796935 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.796883    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8349d596-e27d-46f9-923a-e731adcf5292-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:25.797011 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.796990    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9963d70a-6261-48fc-b1b7-aa3138751293-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.797070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.797017    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9963d70a-6261-48fc-b1b7-aa3138751293-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.797070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.797034    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqk8\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-kube-api-access-wxqk8\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.797070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.797052    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9963d70a-6261-48fc-b1b7-aa3138751293-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.797070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.797065    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9963d70a-6261-48fc-b1b7-aa3138751293-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.842735 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.842713    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:56:25.898121 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898081    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-token\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-token\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898143    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-certs\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898175    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-envoy\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898232    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-kube-api-access-s4zd8\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-kube-api-access-s4zd8\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898300    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-credential-socket\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898323    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5fefae-1b0e-4774-988c-fbe7be25d757-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fefae-1b0e-4774-988c-fbe7be25d757-istiod-ca-cert\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898370    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-data\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898426    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-podinfo\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.898604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898496    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-socket\") pod \"de5fefae-1b0e-4774-988c-fbe7be25d757\" (UID: \"de5fefae-1b0e-4774-988c-fbe7be25d757\") "
Oct 30 20:56:25.899055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.898936    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js49\" (UniqueName: \"kubernetes.io/projected/8349d596-e27d-46f9-923a-e731adcf5292-kube-api-access-5js49\") pod \"payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:25.899055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.899018    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5fefae-1b0e-4774-988c-fbe7be25d757-istiod-ca-cert" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:25.899055 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.899033    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8349d596-e27d-46f9-923a-e731adcf5292-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:25.899216 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.899153    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fefae-1b0e-4774-988c-fbe7be25d757-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:25.899294 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.899273    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-credential-socket" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.900120 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.899563    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-data" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.900120 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.899919    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8349d596-e27d-46f9-923a-e731adcf5292-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:25.900120 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.900048    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-socket" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.900739 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.900715    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-certs" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.900952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.900927    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-envoy" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:25.901416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.901394    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-podinfo" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:56:25.901897 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.901878    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-token" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:25.902425 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.902407    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:56:25.902584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.902523    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-kube-api-access-s4zd8" pod "de5fefae-1b0e-4774-988c-fbe7be25d757" (UID: "de5fefae-1b0e-4774-988c-fbe7be25d757"). InnerVolumeSpecName "kube-api-access-s4zd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:25.958291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:25.958241    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5js49\" (UniqueName: \"kubernetes.io/projected/8349d596-e27d-46f9-923a-e731adcf5292-kube-api-access-5js49\") pod \"payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:26.000831 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000799    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.000971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000834    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.000971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000848    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.000971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000861    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.000971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000874    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.000971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000887    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.000971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000907    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4zd8\" (UniqueName: \"kubernetes.io/projected/de5fefae-1b0e-4774-988c-fbe7be25d757-kube-api-access-s4zd8\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.000971 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.000922    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fefae-1b0e-4774-988c-fbe7be25d757-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:26.015054 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.015031    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:26.254135 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.254110    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"
Oct 30 20:56:26.254135 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.254118    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c" event={"ID":"ada2cf2a-29a4-4042-b8be-9536203e8703","Type":"ContainerDied","Data":"190f81571b2092eaa36bb5559ea73c51b9c2f897448422055507e66bd137228e"}
Oct 30 20:56:26.254340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.254188    5002 scope.go:122] "RemoveContainer" containerID="03d0ba1750e578350b8971ad09407021e614854bd4a913910be9c2502d77b1a8"
Oct 30 20:56:26.256017 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.255997    5002 generic.go:356] "Generic (PLEG): container finished" podID="9963d70a-6261-48fc-b1b7-aa3138751293" containerID="533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498" exitCode=0
Oct 30 20:56:26.256165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.256071    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"
Oct 30 20:56:26.256165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.256096    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" event={"ID":"9963d70a-6261-48fc-b1b7-aa3138751293","Type":"ContainerDied","Data":"533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498"}
Oct 30 20:56:26.256268 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.256162    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh" event={"ID":"9963d70a-6261-48fc-b1b7-aa3138751293","Type":"ContainerDied","Data":"14a017e804464301d66051a6f6a84ce2eefcfbeab4e55abbdf44f7d17f45a643"}
Oct 30 20:56:26.257978 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.257949    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5" event={"ID":"de5fefae-1b0e-4774-988c-fbe7be25d757","Type":"ContainerDied","Data":"f01eb4099491b255a6c427579a354beedfcacb350fa30247f6bb5430cd42c67f"}
Oct 30 20:56:26.258124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.258004    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"
Oct 30 20:56:26.259198 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.259176    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 20:56:26.265996 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.265972    5002 scope.go:122] "RemoveContainer" containerID="533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498"
Oct 30 20:56:26.275164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.275074    5002 scope.go:122] "RemoveContainer" containerID="533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498"
Oct 30 20:56:26.275365 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:26.275346    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498\": container with ID starting with 533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498 not found: ID does not exist" containerID="533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498"
Oct 30 20:56:26.275412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.275372    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498"} err="failed to get container status \"533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498\": rpc error: code = NotFound desc = could not find container \"533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498\": container with ID starting with 533ad6f5166ec9191d30fb5f453e5d9df4b45649678448835e7883c66b163498 not found: ID does not exist"
Oct 30 20:56:26.275412 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.275388    5002 scope.go:122] "RemoveContainer" containerID="fa64e90450ab5c6cad68c33ad20668633fb902bd942814c443e0bf9943eb412d"
Oct 30 20:56:26.396682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.396657    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"]
Oct 30 20:56:26.396789 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:56:26.396747    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8349d596_e27d_46f9_923a_e731adcf5292.slice/crio-7403f83db3a54e76f1ebd19f34d4fc8ec3c49ed138293d6636640463a641fc91 WatchSource:0}: Error finding container 7403f83db3a54e76f1ebd19f34d4fc8ec3c49ed138293d6636640463a641fc91: Status 404 returned error can't find the container with id 7403f83db3a54e76f1ebd19f34d4fc8ec3c49ed138293d6636640463a641fc91
Oct 30 20:56:26.425714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.425690    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"]
Oct 30 20:56:26.430086 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.430065    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5"]
Oct 30 20:56:26.538266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.538211    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"]
Oct 30 20:56:26.541696 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.541668    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh"]
Oct 30 20:56:26.651351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.651326    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"]
Oct 30 20:56:26.655799 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:26.655777    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c"]
Oct 30 20:56:27.119948 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:27.119915    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:27.270907 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:27.270872    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg" event={"ID":"8349d596-e27d-46f9-923a-e731adcf5292","Type":"ContainerStarted","Data":"dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85"}
Oct 30 20:56:27.271088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:27.270948    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg" event={"ID":"8349d596-e27d-46f9-923a-e731adcf5292","Type":"ContainerStarted","Data":"7403f83db3a54e76f1ebd19f34d4fc8ec3c49ed138293d6636640463a641fc91"}
Oct 30 20:56:27.389307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:27.389226    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg" podStartSLOduration=2.389214842 podStartE2EDuration="2.389214842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:27.387701867 +0000 UTC m=+2213.536752856" watchObservedRunningTime="2026-10-30 20:56:27.389214842 +0000 UTC m=+2213.538265828"
Oct 30 20:56:27.969196 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:27.969162    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9963d70a-6261-48fc-b1b7-aa3138751293" path="/var/lib/kubelet/pods/9963d70a-6261-48fc-b1b7-aa3138751293/volumes"
Oct 30 20:56:27.969634 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:27.969620    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada2cf2a-29a4-4042-b8be-9536203e8703" path="/var/lib/kubelet/pods/ada2cf2a-29a4-4042-b8be-9536203e8703/volumes"
Oct 30 20:56:27.970049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:27.970035    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" path="/var/lib/kubelet/pods/de5fefae-1b0e-4774-988c-fbe7be25d757/volumes"
Oct 30 20:56:28.277854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:28.277785    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:30.120137 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.120102    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-cca9f6"]
Oct 30 20:56:30.120914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.120885    5002 cpu_manager.go:414] "containerMap: removing container" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy"
Oct 30 20:56:30.120914 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.120914    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy"
Oct 30 20:56:30.121135 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.121121    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5fefae-1b0e-4774-988c-fbe7be25d757" containerName="istio-proxy"
Oct 30 20:56:30.125876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.125854    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:30.179640 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.179617    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-cca9f6"]
Oct 30 20:56:30.225179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.225156    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vjnsm\""
Oct 30 20:56:30.248106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.248086    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjv9\" (UniqueName: \"kubernetes.io/projected/ab77f241-3c7a-4aad-bdda-d3f82942e398-kube-api-access-7qjv9\") pod \"test-curl-7269-cca9f6\" (UID: \"ab77f241-3c7a-4aad-bdda-d3f82942e398\") " pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:30.349423 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.349395    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjv9\" (UniqueName: \"kubernetes.io/projected/ab77f241-3c7a-4aad-bdda-d3f82942e398-kube-api-access-7qjv9\") pod \"test-curl-7269-cca9f6\" (UID: \"ab77f241-3c7a-4aad-bdda-d3f82942e398\") " pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:30.406707 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.406655    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjv9\" (UniqueName: \"kubernetes.io/projected/ab77f241-3c7a-4aad-bdda-d3f82942e398-kube-api-access-7qjv9\") pod \"test-curl-7269-cca9f6\" (UID: \"ab77f241-3c7a-4aad-bdda-d3f82942e398\") " pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:30.438780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.438760    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:30.819648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:30.819619    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-cca9f6"]
Oct 30 20:56:30.819874 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:56:30.819856    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab77f241_3c7a_4aad_bdda_d3f82942e398.slice/crio-c6552bd3e9d5a184fe4fd1f6c75bd8b09f92ed33498886d1986f4d86e5391b5b WatchSource:0}: Error finding container c6552bd3e9d5a184fe4fd1f6c75bd8b09f92ed33498886d1986f4d86e5391b5b: Status 404 returned error can't find the container with id c6552bd3e9d5a184fe4fd1f6c75bd8b09f92ed33498886d1986f4d86e5391b5b
Oct 30 20:56:31.301005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:31.300966    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-cca9f6" event={"ID":"ab77f241-3c7a-4aad-bdda-d3f82942e398","Type":"ContainerStarted","Data":"732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134"}
Oct 30 20:56:31.301005 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:31.301008    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-cca9f6" event={"ID":"ab77f241-3c7a-4aad-bdda-d3f82942e398","Type":"ContainerStarted","Data":"c6552bd3e9d5a184fe4fd1f6c75bd8b09f92ed33498886d1986f4d86e5391b5b"}
Oct 30 20:56:31.417397 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:31.417343    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-cca9f6" podStartSLOduration=1.417329494 podStartE2EDuration="1.417329494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:31.415501215 +0000 UTC m=+2217.564552202" watchObservedRunningTime="2026-10-30 20:56:31.417329494 +0000 UTC m=+2217.566380481"
Oct 30 20:56:32.149985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:32.149946    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:32.927870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:32.927835    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-cca9f6"]
Oct 30 20:56:32.932300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:32.932271    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-cca9f6"]
Oct 30 20:56:33.314106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:33.314043    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-cca9f6" podUID="ab77f241-3c7a-4aad-bdda-d3f82942e398" containerName="test-curl-7269-cca9f6" containerID="cri-o://732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134" gracePeriod=2
Oct 30 20:56:33.365993 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:33.365959    5002 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-cca9f6\" is forbidden: User \"system:node:ip-10-0-141-29.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-29.us-west-1.compute.internal' and this object" podUID="ab77f241-3c7a-4aad-bdda-d3f82942e398" pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:34.015442 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:34.015406    5002 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-cca9f6\" is forbidden: User \"system:node:ip-10-0-141-29.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-29.us-west-1.compute.internal' and this object" podUID="ab77f241-3c7a-4aad-bdda-d3f82942e398" pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:35.584590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:35.584508    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:35.708480 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:35.708458    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab77f241-3c7a-4aad-bdda-d3f82942e398-kube-api-access-7qjv9\" (UniqueName: \"kubernetes.io/projected/ab77f241-3c7a-4aad-bdda-d3f82942e398-kube-api-access-7qjv9\") pod \"ab77f241-3c7a-4aad-bdda-d3f82942e398\" (UID: \"ab77f241-3c7a-4aad-bdda-d3f82942e398\") "
Oct 30 20:56:35.710576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:35.710552    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab77f241-3c7a-4aad-bdda-d3f82942e398-kube-api-access-7qjv9" pod "ab77f241-3c7a-4aad-bdda-d3f82942e398" (UID: "ab77f241-3c7a-4aad-bdda-d3f82942e398"). InnerVolumeSpecName "kube-api-access-7qjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:35.809602 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:35.809575    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qjv9\" (UniqueName: \"kubernetes.io/projected/ab77f241-3c7a-4aad-bdda-d3f82942e398-kube-api-access-7qjv9\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:35.967654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:35.967626    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab77f241-3c7a-4aad-bdda-d3f82942e398" path="/var/lib/kubelet/pods/ab77f241-3c7a-4aad-bdda-d3f82942e398/volumes"
Oct 30 20:56:36.338576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.338476    5002 generic.go:356] "Generic (PLEG): container finished" podID="ab77f241-3c7a-4aad-bdda-d3f82942e398" containerID="732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134" exitCode=137
Oct 30 20:56:36.338576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.338524    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-cca9f6"
Oct 30 20:56:36.338744 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.338603    5002 scope.go:122] "RemoveContainer" containerID="732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134"
Oct 30 20:56:36.348199 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.348173    5002 scope.go:122] "RemoveContainer" containerID="732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134"
Oct 30 20:56:36.348492 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:36.348472    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134\": container with ID starting with 732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134 not found: ID does not exist" containerID="732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134"
Oct 30 20:56:36.348591 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.348500    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134"} err="failed to get container status \"732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134\": rpc error: code = NotFound desc = could not find container \"732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134\": container with ID starting with 732601af95cb97d8a135c8fc6e7bb2877432e97f7e8ab3787288ad607c7c3134 not found: ID does not exist"
Oct 30 20:56:36.587709 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.587673    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:36.893579 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.893513    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"]
Oct 30 20:56:36.894168 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.894155    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ab77f241-3c7a-4aad-bdda-d3f82942e398" containerName="test-curl-7269-cca9f6"
Oct 30 20:56:36.894215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.894172    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab77f241-3c7a-4aad-bdda-d3f82942e398" containerName="test-curl-7269-cca9f6"
Oct 30 20:56:36.894329 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.894320    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab77f241-3c7a-4aad-bdda-d3f82942e398" containerName="test-curl-7269-cca9f6"
Oct 30 20:56:36.899416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.899396    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:36.953678 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:36.953656    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5dc13e-serving-cert\""
Oct 30 20:56:37.007699 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.007670    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"]
Oct 30 20:56:37.022273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.022249    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.022382 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.022318    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-metrics-certs\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.022442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.022411    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5325fc-fc92-499c-9845-ff8c6303dc31-openshift-service-ca\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.022495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.022472    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsrc\" (UniqueName: \"kubernetes.io/projected/2c5325fc-fc92-499c-9845-ff8c6303dc31-kube-api-access-bcsrc\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.116510 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:37.116464    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:56:37.123243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.123224    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.123362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.123309    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-metrics-certs\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.123362 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.123346    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5325fc-fc92-499c-9845-ff8c6303dc31-openshift-service-ca\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.123464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.123386    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsrc\" (UniqueName: \"kubernetes.io/projected/2c5325fc-fc92-499c-9845-ff8c6303dc31-kube-api-access-bcsrc\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.123464 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:37.123388    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-serving-cert: secret "maas-api-e2e-api-a-5dc13e-serving-cert" not found
Oct 30 20:56:37.123464 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:37.123456    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls podName:2c5325fc-fc92-499c-9845-ff8c6303dc31 nodeName:}" failed. No retries permitted until 2026-10-30 20:56:37.62343521 +0000 UTC m=+2223.772486186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls") pod "maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd" (UID: "2c5325fc-fc92-499c-9845-ff8c6303dc31") : secret "maas-api-e2e-api-a-5dc13e-serving-cert" not found
Oct 30 20:56:37.124143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.124056    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5325fc-fc92-499c-9845-ff8c6303dc31-openshift-service-ca\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.125773 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.125754    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-metrics-certs\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.190270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.190193    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsrc\" (UniqueName: \"kubernetes.io/projected/2c5325fc-fc92-499c-9845-ff8c6303dc31-kube-api-access-bcsrc\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.192087 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.192051    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:37.378876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.378836    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:37.629042 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.629006    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.631761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.631729    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls\") pod \"maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:37.813589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:37.813556    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:38.198441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:38.198412    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"]
Oct 30 20:56:38.198852 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:56:38.198825    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5325fc_fc92_499c_9845_ff8c6303dc31.slice/crio-c1780e4e9d2e3b72baa9f466fcfe389a5e4db4ec03c17fe7b0bdd749def8c5fa WatchSource:0}: Error finding container c1780e4e9d2e3b72baa9f466fcfe389a5e4db4ec03c17fe7b0bdd749def8c5fa: Status 404 returned error can't find the container with id c1780e4e9d2e3b72baa9f466fcfe389a5e4db4ec03c17fe7b0bdd749def8c5fa
Oct 30 20:56:38.362202 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:38.362170    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd" event={"ID":"2c5325fc-fc92-499c-9845-ff8c6303dc31","Type":"ContainerStarted","Data":"c1780e4e9d2e3b72baa9f466fcfe389a5e4db4ec03c17fe7b0bdd749def8c5fa"}
Oct 30 20:56:38.423299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:38.423258    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:38.618690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:38.618626    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:39.287949 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:39.287870    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 20:56:39.370171 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:39.370127    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd" event={"ID":"2c5325fc-fc92-499c-9845-ff8c6303dc31","Type":"ContainerStarted","Data":"b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d"}
Oct 30 20:56:39.370333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:39.370222    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:39.525438 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:39.525381    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd" podStartSLOduration=2.733427948 podStartE2EDuration="3.525361229s" totalImagesPullingTime="791.933281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:39.52247374 +0000 UTC m=+2225.671524727" watchObservedRunningTime="2026-10-30 20:56:39.525361229 +0000 UTC m=+2225.674412216"
Oct 30 20:56:42.287940 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:42.287903    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:45.382845 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:45.382819    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 20:56:45.439614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:45.439571    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"]
Oct 30 20:56:45.547482 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:45.547452    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"]
Oct 30 20:56:45.860136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:45.860100    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:46.047384 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:46.047348    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:46.830610 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:46.830574    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:56:47.242829 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:47.242795    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:50.440818 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.440746    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv" podUID="2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" containerName="payload-processing" containerID="cri-o://69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d" gracePeriod=25
Oct 30 20:56:50.548666 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.548602    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7" podUID="73f8acac-b739-48ad-acfd-67e863e35bc0" containerName="payload-pre-processing" containerID="cri-o://f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441" gracePeriod=25
Oct 30 20:56:50.786231 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.786192    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:56:50.879504 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.879482    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:56:50.900477 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.900457    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f8acac-b739-48ad-acfd-67e863e35bc0-kube-api-access-kqhvp\" (UniqueName: \"kubernetes.io/projected/73f8acac-b739-48ad-acfd-67e863e35bc0-kube-api-access-kqhvp\") pod \"73f8acac-b739-48ad-acfd-67e863e35bc0\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") "
Oct 30 20:56:50.900616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.900495    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-kube-api-access-qv5qw\" (UniqueName: \"kubernetes.io/projected/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-kube-api-access-qv5qw\") pod \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") "
Oct 30 20:56:50.900682 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.900655    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-plugins-config-volume\") pod \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\" (UID: \"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb\") "
Oct 30 20:56:50.900738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.900694    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f8acac-b739-48ad-acfd-67e863e35bc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8acac-b739-48ad-acfd-67e863e35bc0-plugins-config-volume\") pod \"73f8acac-b739-48ad-acfd-67e863e35bc0\" (UID: \"73f8acac-b739-48ad-acfd-67e863e35bc0\") "
Oct 30 20:56:50.901061 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.901036    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-plugins-config-volume" pod "2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" (UID: "2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:50.901137 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.901096    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f8acac-b739-48ad-acfd-67e863e35bc0-plugins-config-volume" pod "73f8acac-b739-48ad-acfd-67e863e35bc0" (UID: "73f8acac-b739-48ad-acfd-67e863e35bc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:50.901224 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.901206    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:50.901283 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.901226    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8acac-b739-48ad-acfd-67e863e35bc0-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:50.902695 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.902673    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-kube-api-access-qv5qw" pod "2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" (UID: "2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb"). InnerVolumeSpecName "kube-api-access-qv5qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:50.903149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:50.903133    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f8acac-b739-48ad-acfd-67e863e35bc0-kube-api-access-kqhvp" pod "73f8acac-b739-48ad-acfd-67e863e35bc0" (UID: "73f8acac-b739-48ad-acfd-67e863e35bc0"). InnerVolumeSpecName "kube-api-access-kqhvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:51.002227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.002194    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqhvp\" (UniqueName: \"kubernetes.io/projected/73f8acac-b739-48ad-acfd-67e863e35bc0-kube-api-access-kqhvp\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:51.002390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.002227    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5qw\" (UniqueName: \"kubernetes.io/projected/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb-kube-api-access-qv5qw\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:51.189322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.189230    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"]
Oct 30 20:56:51.189501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.189452    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn" podUID="064c2f71-0123-4dd2-8dee-7834e2cf7b32" containerName="maas-api" containerID="cri-o://c071454e33b2373f7c73f05c7261040a8f2066c98b375f4688c7a576a3e801a0" gracePeriod=30
Oct 30 20:56:51.339219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.339172    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:51.475598 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.474448    5002 generic.go:356] "Generic (PLEG): container finished" podID="2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" containerID="69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d" exitCode=0
Oct 30 20:56:51.476015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.475683    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv" event={"ID":"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb","Type":"ContainerDied","Data":"69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d"}
Oct 30 20:56:51.476015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.475738    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv" event={"ID":"2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb","Type":"ContainerDied","Data":"c7d11c82154c585ec495dfd539101e3bd2522bfd4ececa2a86ad683a7621a1ed"}
Oct 30 20:56:51.476015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.475766    5002 scope.go:122] "RemoveContainer" containerID="69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d"
Oct 30 20:56:51.476015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.475917    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"
Oct 30 20:56:51.489995 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.489962    5002 generic.go:356] "Generic (PLEG): container finished" podID="064c2f71-0123-4dd2-8dee-7834e2cf7b32" containerID="c071454e33b2373f7c73f05c7261040a8f2066c98b375f4688c7a576a3e801a0" exitCode=0
Oct 30 20:56:51.490140 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.490117    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn" event={"ID":"064c2f71-0123-4dd2-8dee-7834e2cf7b32","Type":"ContainerDied","Data":"c071454e33b2373f7c73f05c7261040a8f2066c98b375f4688c7a576a3e801a0"}
Oct 30 20:56:51.491674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.491647    5002 generic.go:356] "Generic (PLEG): container finished" podID="73f8acac-b739-48ad-acfd-67e863e35bc0" containerID="f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441" exitCode=0
Oct 30 20:56:51.491780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.491679    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7" event={"ID":"73f8acac-b739-48ad-acfd-67e863e35bc0","Type":"ContainerDied","Data":"f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441"}
Oct 30 20:56:51.491780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.491719    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7" event={"ID":"73f8acac-b739-48ad-acfd-67e863e35bc0","Type":"ContainerDied","Data":"17f0fb31ae0c227ed9825c9d730aae824ef163c67ec14b758462f685186544f5"}
Oct 30 20:56:51.491780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.491737    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"
Oct 30 20:56:51.493571 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.493552    5002 scope.go:122] "RemoveContainer" containerID="69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d"
Oct 30 20:56:51.494006 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:51.493979    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d\": container with ID starting with 69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d not found: ID does not exist" containerID="69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d"
Oct 30 20:56:51.494091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.494017    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d"} err="failed to get container status \"69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d\": rpc error: code = NotFound desc = could not find container \"69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d\": container with ID starting with 69a8fad081e04dc2dd8bb9dca0ffabe308e5e7eb105d4b2b1aac528fee90557d not found: ID does not exist"
Oct 30 20:56:51.494091 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.494041    5002 scope.go:122] "RemoveContainer" containerID="f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441"
Oct 30 20:56:51.505490 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.505347    5002 scope.go:122] "RemoveContainer" containerID="f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441"
Oct 30 20:56:51.506018 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:56:51.505993    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441\": container with ID starting with f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441 not found: ID does not exist" containerID="f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441"
Oct 30 20:56:51.506084 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.506027    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441"} err="failed to get container status \"f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441\": rpc error: code = NotFound desc = could not find container \"f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441\": container with ID starting with f66bf5b86ec5c8fef14b5c0da5e4aa69068f2f5b33538087ce196ba057503441 not found: ID does not exist"
Oct 30 20:56:51.566738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.566718    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:56:51.638439 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.638016    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064c2f71-0123-4dd2-8dee-7834e2cf7b32-kube-api-access-fxscb\" (UniqueName: \"kubernetes.io/projected/064c2f71-0123-4dd2-8dee-7834e2cf7b32-kube-api-access-fxscb\") pod \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") "
Oct 30 20:56:51.638733 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.638707    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064c2f71-0123-4dd2-8dee-7834e2cf7b32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c2f71-0123-4dd2-8dee-7834e2cf7b32-openshift-service-ca\") pod \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") "
Oct 30 20:56:51.639100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.639082    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-metrics-certs\") pod \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") "
Oct 30 20:56:51.639882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.639862    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls\") pod \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\" (UID: \"064c2f71-0123-4dd2-8dee-7834e2cf7b32\") "
Oct 30 20:56:51.640740 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.639804    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064c2f71-0123-4dd2-8dee-7834e2cf7b32-openshift-service-ca" pod "064c2f71-0123-4dd2-8dee-7834e2cf7b32" (UID: "064c2f71-0123-4dd2-8dee-7834e2cf7b32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:51.647107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.646774    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-metrics-certs" pod "064c2f71-0123-4dd2-8dee-7834e2cf7b32" (UID: "064c2f71-0123-4dd2-8dee-7834e2cf7b32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:51.647107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.647078    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064c2f71-0123-4dd2-8dee-7834e2cf7b32-kube-api-access-fxscb" pod "064c2f71-0123-4dd2-8dee-7834e2cf7b32" (UID: "064c2f71-0123-4dd2-8dee-7834e2cf7b32"). InnerVolumeSpecName "kube-api-access-fxscb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:51.653332 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.651667    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"]
Oct 30 20:56:51.654869 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.654288    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls" pod "064c2f71-0123-4dd2-8dee-7834e2cf7b32" (UID: "064c2f71-0123-4dd2-8dee-7834e2cf7b32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:51.655650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.655628    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7"]
Oct 30 20:56:51.741802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.741715    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:51.741802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.741744    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c2f71-0123-4dd2-8dee-7834e2cf7b32-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:51.741802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.741760    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxscb\" (UniqueName: \"kubernetes.io/projected/064c2f71-0123-4dd2-8dee-7834e2cf7b32-kube-api-access-fxscb\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:51.741802 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.741774    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c2f71-0123-4dd2-8dee-7834e2cf7b32-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:51.763417 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.763386    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"]
Oct 30 20:56:51.768762 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.768738    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv"]
Oct 30 20:56:51.967643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.967607    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" path="/var/lib/kubelet/pods/2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb/volumes"
Oct 30 20:56:51.968224 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:51.968205    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f8acac-b739-48ad-acfd-67e863e35bc0" path="/var/lib/kubelet/pods/73f8acac-b739-48ad-acfd-67e863e35bc0/volumes"
Oct 30 20:56:52.502934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:52.502902    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"
Oct 30 20:56:52.503755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:52.502901    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn" event={"ID":"064c2f71-0123-4dd2-8dee-7834e2cf7b32","Type":"ContainerDied","Data":"f67a9f302d24a0c284715a0fc51e1bc2ce6dc761d816d7a1ba62e102e2825650"}
Oct 30 20:56:52.503755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:52.503049    5002 scope.go:122] "RemoveContainer" containerID="c071454e33b2373f7c73f05c7261040a8f2066c98b375f4688c7a576a3e801a0"
Oct 30 20:56:52.670372 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:52.670341    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"]
Oct 30 20:56:52.673589 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:52.673570    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-061bda-574cf86c5d-jddfn"]
Oct 30 20:56:52.869784 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:52.869693    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:53.137596 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:53.137508    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:53.322346 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:53.322318    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:53.835979 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:53.835932    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:53.968895 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:53.968847    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064c2f71-0123-4dd2-8dee-7834e2cf7b32" path="/var/lib/kubelet/pods/064c2f71-0123-4dd2-8dee-7834e2cf7b32/volumes"
Oct 30 20:56:56.435786 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.435751    5002 patch_prober.go:29] interesting pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:56:56.436223 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.435829    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:56:56.534665 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.534633    5002 generic.go:356] "Generic (PLEG): container finished" podID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerID="d85cb981d53fc8363179c89b115c1acb69403a8c638a28a090bd0b0de71891f2" exitCode=0
Oct 30 20:56:56.534780 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.534702    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" event={"ID":"88071f20-7d08-4bda-a6c6-65eccb3ab8e7","Type":"ContainerDied","Data":"d85cb981d53fc8363179c89b115c1acb69403a8c638a28a090bd0b0de71891f2"}
Oct 30 20:56:56.535084 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.535066    5002 scope.go:122] "RemoveContainer" containerID="d85cb981d53fc8363179c89b115c1acb69403a8c638a28a090bd0b0de71891f2"
Oct 30 20:56:56.536549 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.536503    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:56.536619 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.536593    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:56.536659 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:56.536630    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:57.546012 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:57.545975    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" event={"ID":"88071f20-7d08-4bda-a6c6-65eccb3ab8e7","Type":"ContainerStarted","Data":"1c4a358103e222e5f47ad1f427b512893b32624110405a55fe9e3485cd7cb8b2"}
Oct 30 20:56:58.030569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:58.029850    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:56:58.030569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:58.029960    5002 patch_prober.go:29] interesting pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:56:58.030569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:58.030015    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:56:58.130558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:58.126887    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:59.030559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:59.024801    5002 patch_prober.go:29] interesting pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:56:59.030559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:59.024899    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:56:59.535554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:59.535084    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:59.731556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:59.726790    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:59.932380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:56:59.932284    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:00.031559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:00.026852    5002 patch_prober.go:29] interesting pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:57:00.031559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:00.026952    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:57:00.353400 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:00.353362    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:01.032555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:01.031802    5002 patch_prober.go:29] interesting pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:57:01.032555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:01.031869    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:57:02.028868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:02.028834    5002 patch_prober.go:29] interesting pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:57:02.029164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:02.029136    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:57:03.027523 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:03.027485    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:57:03.028033 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:03.027856    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:57:03.028603 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:03.028583    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 20:57:03.457318 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:03.457242    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:05.314096 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:05.314054    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"]
Oct 30 20:57:05.364702 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:05.364667    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"]
Oct 30 20:57:05.443888 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:05.443848    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:05.629363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:05.629278    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:05.834713 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:05.834681    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:06.161247 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.161216    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:06.366106 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.366046    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:06.454335 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.454264    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"]
Oct 30 20:57:06.454890 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.454876    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" containerName="payload-processing"
Oct 30 20:57:06.454960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.454891    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" containerName="payload-processing"
Oct 30 20:57:06.454960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.454924    5002 cpu_manager.go:414] "containerMap: removing container" podUID="064c2f71-0123-4dd2-8dee-7834e2cf7b32" containerName="maas-api"
Oct 30 20:57:06.454960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.454929    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064c2f71-0123-4dd2-8dee-7834e2cf7b32" containerName="maas-api"
Oct 30 20:57:06.454960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.454938    5002 cpu_manager.go:414] "containerMap: removing container" podUID="73f8acac-b739-48ad-acfd-67e863e35bc0" containerName="payload-pre-processing"
Oct 30 20:57:06.454960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.454944    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f8acac-b739-48ad-acfd-67e863e35bc0" containerName="payload-pre-processing"
Oct 30 20:57:06.455112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.455047    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="064c2f71-0123-4dd2-8dee-7834e2cf7b32" containerName="maas-api"
Oct 30 20:57:06.455112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.455053    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f8acac-b739-48ad-acfd-67e863e35bc0" containerName="payload-pre-processing"
Oct 30 20:57:06.455112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.455065    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe21dcb-a1a0-4c41-8f9e-b2dc555cd9bb" containerName="payload-processing"
Oct 30 20:57:06.457876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.457859    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.511239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.511202    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-11a0b0-openshift-default-dockercfg-m4bbn\""
Oct 30 20:57:06.564911 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.564876    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"]
Oct 30 20:57:06.575795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.575768    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-certs\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.575948 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.575813    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-credential-socket\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.575948 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.575926    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istiod-ca-cert\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.576095 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.576071    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-envoy\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.576147 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.576134    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-podinfo\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.576189 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.576161    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-socket\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.576267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.576252    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-data\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.576323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.576289    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-token\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.576405 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.576387    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjssf\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-kube-api-access-gjssf\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.600806 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.600775    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:06.677652 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.677601    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjssf\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-kube-api-access-gjssf\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.677834 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.677702    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-certs\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.677834 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.677741    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-credential-socket\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.677834 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.677792    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istiod-ca-cert\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.677998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.677913    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-envoy\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.677998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.677957    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-podinfo\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.677998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.677988    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-socket\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.678166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.678022    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-data\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.678166 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.678056    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-token\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.678307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.678274    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-certs\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.678575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.678553    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-credential-socket\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.678955 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.678933    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-socket\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.679158 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.679137    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istiod-ca-cert\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.679382 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.679365    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-data\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.681968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.681932    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-podinfo\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.683039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.683019    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-envoy\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.741553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.738260    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-token\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.742190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.742160    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjssf\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-kube-api-access-gjssf\") pod \"e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") " pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.770567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.770514    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:06.798759 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:06.798725    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:07.115285 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:57:07.115204    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:57:07.207704 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:57:07.207671    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b5bd60_b2cb_4fbb_8b5b_f93f166dd64f.slice/crio-d7daa2d7e86b86aa907e74906d425ba29749405cbee24b83f1b30e9db2067591 WatchSource:0}: Error finding container d7daa2d7e86b86aa907e74906d425ba29749405cbee24b83f1b30e9db2067591: Status 404 returned error can't find the container with id d7daa2d7e86b86aa907e74906d425ba29749405cbee24b83f1b30e9db2067591
Oct 30 20:57:07.208813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.208792    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"]
Oct 30 20:57:07.210038 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.210003    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:07.210121 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.210071    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:07.210121 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.210102    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:07.634686 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.634566    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" event={"ID":"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f","Type":"ContainerStarted","Data":"e45614f34112a5c67e2aec0fb098cc08e6ea342cbb92aa2eceaf9eedc2e9cce3"}
Oct 30 20:57:07.634686 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.634626    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" event={"ID":"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f","Type":"ContainerStarted","Data":"d7daa2d7e86b86aa907e74906d425ba29749405cbee24b83f1b30e9db2067591"}
Oct 30 20:57:07.759165 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.759107    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" podStartSLOduration=1.7590912950000002 podStartE2EDuration="1.759091295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:07.755829094 +0000 UTC m=+2253.904880093" watchObservedRunningTime="2026-10-30 20:57:07.759091295 +0000 UTC m=+2253.908142281"
Oct 30 20:57:07.771111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:07.771085    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:08.627466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:08.627428    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:08.778138 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:08.778103    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:09.661194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:09.661155    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:09.661980 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:09.661957    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:57:10.549181 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:10.549145    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:10.739493 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:10.739459    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:10.922996 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:10.922924    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:12.758521 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:12.758483    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:12.956769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:12.956733    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:13.935902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:13.935864    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:16.292967 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.292931    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:16.465804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.465773    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:16.663734 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.663662    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"]
Oct 30 20:57:16.667146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.667131    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:16.781782 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.781744    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"]
Oct 30 20:57:16.782094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.782073    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-11a0b0\""
Oct 30 20:57:16.782307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.782292    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-11a0b0-dockercfg-fhbtb\""
Oct 30 20:57:16.808668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.808615    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrxgh\" (UniqueName: \"kubernetes.io/projected/c9725de7-41ec-4ebd-bde8-f9fe5c741209-kube-api-access-rrxgh\") pod \"payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:16.809501 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.809469    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9725de7-41ec-4ebd-bde8-f9fe5c741209-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:16.837798 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.837773    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"]
Oct 30 20:57:16.842129 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.842108    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:16.874926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.874899    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"]
Oct 30 20:57:16.911441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.911407    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxgh\" (UniqueName: \"kubernetes.io/projected/c9725de7-41ec-4ebd-bde8-f9fe5c741209-kube-api-access-rrxgh\") pod \"payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:16.911606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.911470    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9725de7-41ec-4ebd-bde8-f9fe5c741209-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:16.912333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.912300    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9725de7-41ec-4ebd-bde8-f9fe5c741209-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:16.968608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.968584    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrxgh\" (UniqueName: \"kubernetes.io/projected/c9725de7-41ec-4ebd-bde8-f9fe5c741209-kube-api-access-rrxgh\") pod \"payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:16.980998 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:16.980973    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:17.012982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.012953    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpkbz\" (UniqueName: \"kubernetes.io/projected/c784da18-e7c2-4288-b725-7ca1b42450e2-kube-api-access-mpkbz\") pod \"payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:17.013097 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.013011    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784da18-e7c2-4288-b725-7ca1b42450e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:17.115418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.114874    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkbz\" (UniqueName: \"kubernetes.io/projected/c784da18-e7c2-4288-b725-7ca1b42450e2-kube-api-access-mpkbz\") pod \"payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:17.115418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.114951    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784da18-e7c2-4288-b725-7ca1b42450e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:17.115734 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.115710    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784da18-e7c2-4288-b725-7ca1b42450e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:17.172592 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.172565    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkbz\" (UniqueName: \"kubernetes.io/projected/c784da18-e7c2-4288-b725-7ca1b42450e2-kube-api-access-mpkbz\") pod \"payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:17.366305 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.366283    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"]
Oct 30 20:57:17.366905 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:57:17.366881    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9725de7_41ec_4ebd_bde8_f9fe5c741209.slice/crio-aae81923f71e9721149a0ea0ee89736e6226f497709eb5b2d8b034d7fa43e3bc WatchSource:0}: Error finding container aae81923f71e9721149a0ea0ee89736e6226f497709eb5b2d8b034d7fa43e3bc: Status 404 returned error can't find the container with id aae81923f71e9721149a0ea0ee89736e6226f497709eb5b2d8b034d7fa43e3bc
Oct 30 20:57:17.460299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.460276    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:17.725855 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.725816    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" event={"ID":"c9725de7-41ec-4ebd-bde8-f9fe5c741209","Type":"ContainerStarted","Data":"85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf"}
Oct 30 20:57:17.726050 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.725886    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" event={"ID":"c9725de7-41ec-4ebd-bde8-f9fe5c741209","Type":"ContainerStarted","Data":"aae81923f71e9721149a0ea0ee89736e6226f497709eb5b2d8b034d7fa43e3bc"}
Oct 30 20:57:17.726136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.726107    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:17.847508 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.847482    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"]
Oct 30 20:57:17.850447 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:57:17.850424    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc784da18_e7c2_4288_b725_7ca1b42450e2.slice/crio-f57ee6897c7fb02d3b5153cab98ba0a37a1fcf0fe3b2d8b65c206d4bf5526e8e WatchSource:0}: Error finding container f57ee6897c7fb02d3b5153cab98ba0a37a1fcf0fe3b2d8b65c206d4bf5526e8e: Status 404 returned error can't find the container with id f57ee6897c7fb02d3b5153cab98ba0a37a1fcf0fe3b2d8b65c206d4bf5526e8e
Oct 30 20:57:17.851486 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:17.851183    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" podStartSLOduration=1.8511674409999999 podStartE2EDuration="1.851167441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:17.845136715 +0000 UTC m=+2263.994187703" watchObservedRunningTime="2026-10-30 20:57:17.851167441 +0000 UTC m=+2264.000218430"
Oct 30 20:57:18.693638 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:18.693603    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:18.737173 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:18.737140    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh" event={"ID":"c784da18-e7c2-4288-b725-7ca1b42450e2","Type":"ContainerStarted","Data":"cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a"}
Oct 30 20:57:18.737365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:18.737233    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh" event={"ID":"c784da18-e7c2-4288-b725-7ca1b42450e2","Type":"ContainerStarted","Data":"f57ee6897c7fb02d3b5153cab98ba0a37a1fcf0fe3b2d8b65c206d4bf5526e8e"}
Oct 30 20:57:18.737365 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:18.737348    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:18.857932 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:18.857883    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh" podStartSLOduration=2.857869752 podStartE2EDuration="2.857869752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:18.854307281 +0000 UTC m=+2265.003358269" watchObservedRunningTime="2026-10-30 20:57:18.857869752 +0000 UTC m=+2265.006920738"
Oct 30 20:57:19.223225 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:19.223194    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:20.335630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.335520    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main" containerID="cri-o://dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050" gracePeriod=45
Oct 30 20:57:20.385618 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.385572    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main" containerID="cri-o://42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408" gracePeriod=45
Oct 30 20:57:20.693275 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.693251    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:57:20.700695 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.700673    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:57:20.756605 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.756573    5002 generic.go:356] "Generic (PLEG): container finished" podID="ae4363cb-f28c-4534-9800-b51e810117fa" containerID="42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408" exitCode=0
Oct 30 20:57:20.756753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.756630    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" event={"ID":"ae4363cb-f28c-4534-9800-b51e810117fa","Type":"ContainerDied","Data":"42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408"}
Oct 30 20:57:20.756753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.756684    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk" event={"ID":"ae4363cb-f28c-4534-9800-b51e810117fa","Type":"ContainerDied","Data":"173584537b0079f30fe6a4c2bbff5cedb3aac86b1c4e3638c420a05489be91e2"}
Oct 30 20:57:20.756753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.756706    5002 scope.go:122] "RemoveContainer" containerID="42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408"
Oct 30 20:57:20.757227 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.757037    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"
Oct 30 20:57:20.759691 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.759672    5002 generic.go:356] "Generic (PLEG): container finished" podID="231ea425-b247-4ce1-a74f-18354c227a8f" containerID="dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050" exitCode=0
Oct 30 20:57:20.759795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.759708    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"
Oct 30 20:57:20.759795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.759743    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" event={"ID":"231ea425-b247-4ce1-a74f-18354c227a8f","Type":"ContainerDied","Data":"dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050"}
Oct 30 20:57:20.759795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.759766    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg" event={"ID":"231ea425-b247-4ce1-a74f-18354c227a8f","Type":"ContainerDied","Data":"c0acb2de4344d32e4770aa2b8834845283fc24120dcd20ebe1a06797c8cd04a6"}
Oct 30 20:57:20.770951 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.770934    5002 scope.go:122] "RemoveContainer" containerID="42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408"
Oct 30 20:57:20.771240 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:57:20.771222    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408\": container with ID starting with 42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408 not found: ID does not exist" containerID="42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408"
Oct 30 20:57:20.771337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.771251    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408"} err="failed to get container status \"42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408\": rpc error: code = NotFound desc = could not find container \"42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408\": container with ID starting with 42c0e6d16cde63b7597d9ac8a2d33467e7ffe3949654f3996d576f53f9873408 not found: ID does not exist"
Oct 30 20:57:20.771337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.771274    5002 scope.go:122] "RemoveContainer" containerID="dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050"
Oct 30 20:57:20.782864 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.782843    5002 scope.go:122] "RemoveContainer" containerID="dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050"
Oct 30 20:57:20.783174 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:57:20.783143    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050\": container with ID starting with dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050 not found: ID does not exist" containerID="dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050"
Oct 30 20:57:20.783266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.783179    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050"} err="failed to get container status \"dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050\": rpc error: code = NotFound desc = could not find container \"dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050\": container with ID starting with dcec0b9fe6826a1e4eda8e1173092244fb2feab5ef1d8286278786a5ada43050 not found: ID does not exist"
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.858692    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231ea425-b247-4ce1-a74f-18354c227a8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/231ea425-b247-4ce1-a74f-18354c227a8f-tls-certs\") pod \"231ea425-b247-4ce1-a74f-18354c227a8f\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.858742    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-tmp-dir\") pod \"231ea425-b247-4ce1-a74f-18354c227a8f\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.858810    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-home\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-home\") pod \"ae4363cb-f28c-4534-9800-b51e810117fa\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.858849    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4363cb-f28c-4534-9800-b51e810117fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4363cb-f28c-4534-9800-b51e810117fa-tls-certs\") pod \"ae4363cb-f28c-4534-9800-b51e810117fa\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.858884    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-dshm\") pod \"231ea425-b247-4ce1-a74f-18354c227a8f\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.858915    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-dshm\") pod \"ae4363cb-f28c-4534-9800-b51e810117fa\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.858971    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4363cb-f28c-4534-9800-b51e810117fa-kube-api-access-qwm6x\" (UniqueName: \"kubernetes.io/projected/ae4363cb-f28c-4534-9800-b51e810117fa-kube-api-access-qwm6x\") pod \"ae4363cb-f28c-4534-9800-b51e810117fa\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.859042    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-home\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-home\") pod \"231ea425-b247-4ce1-a74f-18354c227a8f\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.859186    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-model-cache\") pod \"231ea425-b247-4ce1-a74f-18354c227a8f\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.859248    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-tmp-dir\") pod \"ae4363cb-f28c-4534-9800-b51e810117fa\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.859372    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231ea425-b247-4ce1-a74f-18354c227a8f-kube-api-access-jj7xv\" (UniqueName: \"kubernetes.io/projected/231ea425-b247-4ce1-a74f-18354c227a8f-kube-api-access-jj7xv\") pod \"231ea425-b247-4ce1-a74f-18354c227a8f\" (UID: \"231ea425-b247-4ce1-a74f-18354c227a8f\") "
Oct 30 20:57:20.859560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.859421    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-model-cache\") pod \"ae4363cb-f28c-4534-9800-b51e810117fa\" (UID: \"ae4363cb-f28c-4534-9800-b51e810117fa\") "
Oct 30 20:57:20.864555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.861214    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-tmp-dir" pod "231ea425-b247-4ce1-a74f-18354c227a8f" (UID: "231ea425-b247-4ce1-a74f-18354c227a8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.864555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.861509    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-home" pod "231ea425-b247-4ce1-a74f-18354c227a8f" (UID: "231ea425-b247-4ce1-a74f-18354c227a8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.864555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.861519    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-tmp-dir" pod "ae4363cb-f28c-4534-9800-b51e810117fa" (UID: "ae4363cb-f28c-4534-9800-b51e810117fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.864555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.861732    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-home" pod "ae4363cb-f28c-4534-9800-b51e810117fa" (UID: "ae4363cb-f28c-4534-9800-b51e810117fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.864555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.861738    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-model-cache" pod "231ea425-b247-4ce1-a74f-18354c227a8f" (UID: "231ea425-b247-4ce1-a74f-18354c227a8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.864555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.863358    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-model-cache" pod "ae4363cb-f28c-4534-9800-b51e810117fa" (UID: "ae4363cb-f28c-4534-9800-b51e810117fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.866023 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.865996    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4363cb-f28c-4534-9800-b51e810117fa-tls-certs" pod "ae4363cb-f28c-4534-9800-b51e810117fa" (UID: "ae4363cb-f28c-4534-9800-b51e810117fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:20.866710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.866672    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231ea425-b247-4ce1-a74f-18354c227a8f-kube-api-access-jj7xv" pod "231ea425-b247-4ce1-a74f-18354c227a8f" (UID: "231ea425-b247-4ce1-a74f-18354c227a8f"). InnerVolumeSpecName "kube-api-access-jj7xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:20.870812 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.870778    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-dshm" pod "ae4363cb-f28c-4534-9800-b51e810117fa" (UID: "ae4363cb-f28c-4534-9800-b51e810117fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.871000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.870916    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231ea425-b247-4ce1-a74f-18354c227a8f-tls-certs" pod "231ea425-b247-4ce1-a74f-18354c227a8f" (UID: "231ea425-b247-4ce1-a74f-18354c227a8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:20.871093 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.871044    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-dshm" pod "231ea425-b247-4ce1-a74f-18354c227a8f" (UID: "231ea425-b247-4ce1-a74f-18354c227a8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.872314 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.872292    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4363cb-f28c-4534-9800-b51e810117fa-kube-api-access-qwm6x" pod "ae4363cb-f28c-4534-9800-b51e810117fa" (UID: "ae4363cb-f28c-4534-9800-b51e810117fa"). InnerVolumeSpecName "kube-api-access-qwm6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:20.961413 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961387    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961432    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961443    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7xv\" (UniqueName: \"kubernetes.io/projected/231ea425-b247-4ce1-a74f-18354c227a8f-kube-api-access-jj7xv\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961453    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961461    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231ea425-b247-4ce1-a74f-18354c227a8f-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961469    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961476    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961484    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4363cb-f28c-4534-9800-b51e810117fa-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961491    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961497    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4363cb-f28c-4534-9800-b51e810117fa-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961505    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwm6x\" (UniqueName: \"kubernetes.io/projected/ae4363cb-f28c-4534-9800-b51e810117fa-kube-api-access-qwm6x\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.961590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:20.961512    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231ea425-b247-4ce1-a74f-18354c227a8f-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:21.181748 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:21.181715    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"]
Oct 30 20:57:21.186487 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:21.186464    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/premium-w1-5f9b95-kserve-588d4dcd5f-t8wtk"]
Oct 30 20:57:21.297001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:21.296934    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"]
Oct 30 20:57:21.302273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:21.302247    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-2-w1-5f9b95-kserve-65cc5bc8f7-z4jrg"]
Oct 30 20:57:21.569944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:21.569807    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:21.971315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:21.971278    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" path="/var/lib/kubelet/pods/231ea425-b247-4ce1-a74f-18354c227a8f/volumes"
Oct 30 20:57:21.972007 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:21.971988    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" path="/var/lib/kubelet/pods/ae4363cb-f28c-4534-9800-b51e810117fa/volumes"
Oct 30 20:57:23.614574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.614540    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"]
Oct 30 20:57:23.665305 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.665273    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"]
Oct 30 20:57:23.669447 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.669417    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"]
Oct 30 20:57:23.669626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.669597    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68" podUID="772b740d-1836-4496-b4b9-9ff6f1a58b4d" containerName="maas-api" containerID="cri-o://481e2294bd309b10be3e37da85a5affd438e801906465d8f0fa73e1bdc967372" gracePeriod=30
Oct 30 20:57:23.790910 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.790882    5002 generic.go:356] "Generic (PLEG): container finished" podID="772b740d-1836-4496-b4b9-9ff6f1a58b4d" containerID="481e2294bd309b10be3e37da85a5affd438e801906465d8f0fa73e1bdc967372" exitCode=0
Oct 30 20:57:23.791040 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.790950    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68" event={"ID":"772b740d-1836-4496-b4b9-9ff6f1a58b4d","Type":"ContainerDied","Data":"481e2294bd309b10be3e37da85a5affd438e801906465d8f0fa73e1bdc967372"}
Oct 30 20:57:23.795764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.795738    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:23.977143 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.977119    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:57:23.980511 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:23.980480    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:24.099463 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.099436    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772b740d-1836-4496-b4b9-9ff6f1a58b4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772b740d-1836-4496-b4b9-9ff6f1a58b4d-openshift-service-ca\") pod \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") "
Oct 30 20:57:24.099667 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.099604    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-metrics-certs\") pod \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") "
Oct 30 20:57:24.099769 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.099685    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls\") pod \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") "
Oct 30 20:57:24.099836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.099765    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772b740d-1836-4496-b4b9-9ff6f1a58b4d-kube-api-access-4kpj9\" (UniqueName: \"kubernetes.io/projected/772b740d-1836-4496-b4b9-9ff6f1a58b4d-kube-api-access-4kpj9\") pod \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\" (UID: \"772b740d-1836-4496-b4b9-9ff6f1a58b4d\") "
Oct 30 20:57:24.099893 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.099836    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772b740d-1836-4496-b4b9-9ff6f1a58b4d-openshift-service-ca" pod "772b740d-1836-4496-b4b9-9ff6f1a58b4d" (UID: "772b740d-1836-4496-b4b9-9ff6f1a58b4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:24.100386 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.100362    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772b740d-1836-4496-b4b9-9ff6f1a58b4d-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:24.116884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.107204    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772b740d-1836-4496-b4b9-9ff6f1a58b4d-kube-api-access-4kpj9" pod "772b740d-1836-4496-b4b9-9ff6f1a58b4d" (UID: "772b740d-1836-4496-b4b9-9ff6f1a58b4d"). InnerVolumeSpecName "kube-api-access-4kpj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:24.116884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.108853    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-metrics-certs" pod "772b740d-1836-4496-b4b9-9ff6f1a58b4d" (UID: "772b740d-1836-4496-b4b9-9ff6f1a58b4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:24.116884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.113857    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls" pod "772b740d-1836-4496-b4b9-9ff6f1a58b4d" (UID: "772b740d-1836-4496-b4b9-9ff6f1a58b4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:24.202027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.201956    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:24.202027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.201984    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772b740d-1836-4496-b4b9-9ff6f1a58b4d-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:24.202027 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.201994    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kpj9\" (UniqueName: \"kubernetes.io/projected/772b740d-1836-4496-b4b9-9ff6f1a58b4d-kube-api-access-4kpj9\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:24.799924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.799882    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"
Oct 30 20:57:24.799924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.799905    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68" event={"ID":"772b740d-1836-4496-b4b9-9ff6f1a58b4d","Type":"ContainerDied","Data":"95d65d85662de56af26cd3f8dfc25a030704deecb49e0859a64b6d36a5c259e1"}
Oct 30 20:57:24.800372 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.799967    5002 scope.go:122] "RemoveContainer" containerID="481e2294bd309b10be3e37da85a5affd438e801906465d8f0fa73e1bdc967372"
Oct 30 20:57:24.968543 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.968513    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"]
Oct 30 20:57:24.974002 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:24.973973    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27219c73-77cb7fc6f4-v7x68"]
Oct 30 20:57:25.398263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.398232    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"]
Oct 30 20:57:25.398853 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.398840    5002 cpu_manager.go:414] "containerMap: removing container" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main"
Oct 30 20:57:25.398909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.398854    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main"
Oct 30 20:57:25.398909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.398886    5002 cpu_manager.go:414] "containerMap: removing container" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main"
Oct 30 20:57:25.398909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.398901    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main"
Oct 30 20:57:25.399001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.398916    5002 cpu_manager.go:414] "containerMap: removing container" podUID="772b740d-1836-4496-b4b9-9ff6f1a58b4d" containerName="maas-api"
Oct 30 20:57:25.399001 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.398921    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772b740d-1836-4496-b4b9-9ff6f1a58b4d" containerName="maas-api"
Oct 30 20:57:25.399063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.399006    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="231ea425-b247-4ce1-a74f-18354c227a8f" containerName="main"
Oct 30 20:57:25.399063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.399016    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="772b740d-1836-4496-b4b9-9ff6f1a58b4d" containerName="maas-api"
Oct 30 20:57:25.399063 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.399025    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4363cb-f28c-4534-9800-b51e810117fa" containerName="main"
Oct 30 20:57:25.401729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.401709    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.455609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.455586    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cd2d96-serving-cert\""
Oct 30 20:57:25.461148 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.461119    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"]
Oct 30 20:57:25.492173 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.492129    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:25.517557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.517505    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtck\" (UniqueName: \"kubernetes.io/projected/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-kube-api-access-nhtck\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.517719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.517633    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.517719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.517688    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.517719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.517705    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-metrics-certs\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.619300 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.619262    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.619462 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.619380    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.619462 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.619416    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-metrics-certs\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.619462 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:57:25.619442    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-serving-cert: secret "maas-api-e2e-api-a-cd2d96-serving-cert" not found
Oct 30 20:57:25.619656 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:57:25.619518    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls podName:75b1e9f0-85ae-4b8c-82d0-85eda9315a07 nodeName:}" failed. No retries permitted until 2026-10-30 20:57:26.119496607 +0000 UTC m=+2272.268547575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls") pod "maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7" (UID: "75b1e9f0-85ae-4b8c-82d0-85eda9315a07") : secret "maas-api-e2e-api-a-cd2d96-serving-cert" not found
Oct 30 20:57:25.619656 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.619587    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtck\" (UniqueName: \"kubernetes.io/projected/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-kube-api-access-nhtck\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.620206 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.620184    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.622252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.622228    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-metrics-certs\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.670009 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.669923    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:25.680124 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.680096    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtck\" (UniqueName: \"kubernetes.io/projected/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-kube-api-access-nhtck\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:25.842722 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.842690    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:25.990714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:25.990189    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772b740d-1836-4496-b4b9-9ff6f1a58b4d" path="/var/lib/kubelet/pods/772b740d-1836-4496-b4b9-9ff6f1a58b4d/volumes"
Oct 30 20:57:26.019793 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.019754    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:26.127608 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.127569    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:26.130136 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.130115    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:26.287421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.287351    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:26.314456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.314432    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:26.461661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.461632    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:26.691972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.691951    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"]
Oct 30 20:57:26.693966 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:57:26.693937    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b1e9f0_85ae_4b8c_82d0_85eda9315a07.slice/crio-1bdef8ae7aad713415ed54bd5cc8c1731a702c5a2b11746d6160f7482fda8afb WatchSource:0}: Error finding container 1bdef8ae7aad713415ed54bd5cc8c1731a702c5a2b11746d6160f7482fda8afb: Status 404 returned error can't find the container with id 1bdef8ae7aad713415ed54bd5cc8c1731a702c5a2b11746d6160f7482fda8afb
Oct 30 20:57:26.822342 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:26.822314    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7" event={"ID":"75b1e9f0-85ae-4b8c-82d0-85eda9315a07","Type":"ContainerStarted","Data":"1bdef8ae7aad713415ed54bd5cc8c1731a702c5a2b11746d6160f7482fda8afb"}
Oct 30 20:57:27.830595 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:27.830514    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7" event={"ID":"75b1e9f0-85ae-4b8c-82d0-85eda9315a07","Type":"ContainerStarted","Data":"773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c"}
Oct 30 20:57:27.831008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:27.830616    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:27.947292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:27.947205    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7" podStartSLOduration=2.025492198 podStartE2EDuration="2.947193234s" totalImagesPullingTime="921.701036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:27.947060365 +0000 UTC m=+2274.096111352" watchObservedRunningTime="2026-10-30 20:57:27.947193234 +0000 UTC m=+2274.096244219"
Oct 30 20:57:28.741191 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:28.741165    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 20:57:29.415389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:29.415357    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"]
Oct 30 20:57:29.415826 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:29.415543    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" containerID="cri-o://ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095" gracePeriod=30
Oct 30 20:57:29.749013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:29.748989    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 20:57:30.895928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:30.895887    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:31.491875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:31.491801    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:31.574992 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:31.574960    5002 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:57:31.575125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:31.575031    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:57:33.611754 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:33.611719    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"]
Oct 30 20:57:33.723299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:33.723272    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"]
Oct 30 20:57:33.844309 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:33.844278    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 20:57:34.018934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.018899    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:34.198898 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.198861    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:34.700081 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.700058    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:57:34.721884 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.721859    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-socket\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.721986 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.721950    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istiod-ca-cert\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.721986 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.721971    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-envoy\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.722058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.721992    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-certs\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.722092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722057    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-credential-socket\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.722092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722067    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-socket" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:34.722161 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722089    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-kube-api-access-bjnzg\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-kube-api-access-bjnzg\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.722205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-token\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-token\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.722205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722183    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-podinfo\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.722306 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722278    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-data\") pod \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\" (UID: \"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71\") "
Oct 30 20:57:34.722363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722308    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-credential-socket" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:34.722363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722331    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-certs" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:34.722363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722352    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istiod-ca-cert" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:34.722643 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722619    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-data" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:34.722993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722944    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.722993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722968    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.722993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722981    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.723204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.722995    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.723204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.723008    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.724380 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.724354    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-envoy" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:34.724466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.724420    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-token" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:34.724466 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.724444    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-kube-api-access-bjnzg" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "kube-api-access-bjnzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:34.724612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.724460    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-podinfo" pod "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" (UID: "cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:57:34.824461 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.824437    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.824604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.824464    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjnzg\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-kube-api-access-bjnzg\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.824604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.824475    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.824604 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.824483    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.884684 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.884645    5002 generic.go:356] "Generic (PLEG): container finished" podID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerID="ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095" exitCode=0
Oct 30 20:57:34.884830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.884685    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"
Oct 30 20:57:34.884830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.884698    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" event={"ID":"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71","Type":"ContainerDied","Data":"ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095"}
Oct 30 20:57:34.884830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.884785    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2" event={"ID":"cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71","Type":"ContainerDied","Data":"92d4b35edde33c2a5edb2fbc4588d7630779cb4a01666ea89f86fcb1f780e5b2"}
Oct 30 20:57:34.884830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.884822    5002 scope.go:122] "RemoveContainer" containerID="ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095"
Oct 30 20:57:34.895845 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.895829    5002 scope.go:122] "RemoveContainer" containerID="ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095"
Oct 30 20:57:34.896107 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:57:34.896089    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095\": container with ID starting with ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095 not found: ID does not exist" containerID="ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095"
Oct 30 20:57:34.896192 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:34.896113    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095"} err="failed to get container status \"ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095\": rpc error: code = NotFound desc = could not find container \"ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095\": container with ID starting with ae90b43b30397bdc94801969c18c5941a0fd06c813b3e8ea4f5e3d24cb8bf095 not found: ID does not exist"
Oct 30 20:57:35.055327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:35.055301    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"]
Oct 30 20:57:35.063508 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:35.063474    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2"]
Oct 30 20:57:35.966989 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:35.966956    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" path="/var/lib/kubelet/pods/cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71/volumes"
Oct 30 20:57:36.087558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:36.087518    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:36.104436 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:36.104406    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"]
Oct 30 20:57:36.216182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:36.216149    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"]
Oct 30 20:57:36.570242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:36.570205    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:36.748481 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:36.748449    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:36.922373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:36.922283    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:37.117721 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:57:37.117684    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:57:38.612964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.612924    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h" podUID="46a5c650-1c2b-49da-b4f0-74ef63f02530" containerName="payload-processing" containerID="cri-o://b379cdb9b66412cfca094a4f5f72b438c4e59d26de06d4b961b0db577238e534" gracePeriod=25
Oct 30 20:57:38.638983 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.638950    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main" containerID="cri-o://b1864d37532fddd56be5d74fc98e867e9d58a0434c1f7aec28642c8e5a3bbdb0" gracePeriod=45
Oct 30 20:57:38.686926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.686897    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main" containerID="cri-o://9e8150813c2419f6712694f013b0c07cdaf2b38142ce101a338441d4b00e7689" gracePeriod=45
Oct 30 20:57:38.724276 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.724248    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm" podUID="441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" containerName="payload-pre-processing" containerID="cri-o://aa94307b74bc3a349afac548bf08a1b475fa8681516de38c1af45716f428e1b3" gracePeriod=25
Oct 30 20:57:38.736922 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.736880    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 30 20:57:38.917713 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.917506    5002 generic.go:356] "Generic (PLEG): container finished" podID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerID="b1864d37532fddd56be5d74fc98e867e9d58a0434c1f7aec28642c8e5a3bbdb0" exitCode=0
Oct 30 20:57:38.917713 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.917654    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" event={"ID":"f916d575-cbfd-4e99-9b00-e6d70dfdd872","Type":"ContainerDied","Data":"b1864d37532fddd56be5d74fc98e867e9d58a0434c1f7aec28642c8e5a3bbdb0"}
Oct 30 20:57:38.919847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.919463    5002 generic.go:356] "Generic (PLEG): container finished" podID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerID="9e8150813c2419f6712694f013b0c07cdaf2b38142ce101a338441d4b00e7689" exitCode=0
Oct 30 20:57:38.919847 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.919577    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" event={"ID":"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a","Type":"ContainerDied","Data":"9e8150813c2419f6712694f013b0c07cdaf2b38142ce101a338441d4b00e7689"}
Oct 30 20:57:38.921456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.921434    5002 generic.go:356] "Generic (PLEG): container finished" podID="441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" containerID="aa94307b74bc3a349afac548bf08a1b475fa8681516de38c1af45716f428e1b3" exitCode=0
Oct 30 20:57:38.921600 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.921576    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm" event={"ID":"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19","Type":"ContainerDied","Data":"aa94307b74bc3a349afac548bf08a1b475fa8681516de38c1af45716f428e1b3"}
Oct 30 20:57:38.923401 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.923261    5002 generic.go:356] "Generic (PLEG): container finished" podID="46a5c650-1c2b-49da-b4f0-74ef63f02530" containerID="b379cdb9b66412cfca094a4f5f72b438c4e59d26de06d4b961b0db577238e534" exitCode=0
Oct 30 20:57:38.923401 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.923355    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h" event={"ID":"46a5c650-1c2b-49da-b4f0-74ef63f02530","Type":"ContainerDied","Data":"b379cdb9b66412cfca094a4f5f72b438c4e59d26de06d4b961b0db577238e534"}
Oct 30 20:57:38.984924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:38.984899    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:57:39.089518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.089491    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a5c650-1c2b-49da-b4f0-74ef63f02530-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a5c650-1c2b-49da-b4f0-74ef63f02530-plugins-config-volume\") pod \"46a5c650-1c2b-49da-b4f0-74ef63f02530\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") "
Oct 30 20:57:39.089686 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.089662    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a5c650-1c2b-49da-b4f0-74ef63f02530-plugins-config-volume" pod "46a5c650-1c2b-49da-b4f0-74ef63f02530" (UID: "46a5c650-1c2b-49da-b4f0-74ef63f02530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:39.089753 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.089702    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a5c650-1c2b-49da-b4f0-74ef63f02530-kube-api-access-t9bgb\" (UniqueName: \"kubernetes.io/projected/46a5c650-1c2b-49da-b4f0-74ef63f02530-kube-api-access-t9bgb\") pod \"46a5c650-1c2b-49da-b4f0-74ef63f02530\" (UID: \"46a5c650-1c2b-49da-b4f0-74ef63f02530\") "
Oct 30 20:57:39.091112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.090921    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a5c650-1c2b-49da-b4f0-74ef63f02530-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.095403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.095380    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a5c650-1c2b-49da-b4f0-74ef63f02530-kube-api-access-t9bgb" pod "46a5c650-1c2b-49da-b4f0-74ef63f02530" (UID: "46a5c650-1c2b-49da-b4f0-74ef63f02530"). InnerVolumeSpecName "kube-api-access-t9bgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.101817 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.101793    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:57:39.119614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.119589    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:57:39.125074 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.125016    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:57:39.191788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.191758    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-plugins-config-volume\") pod \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") "
Oct 30 20:57:39.191969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.191802    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f916d575-cbfd-4e99-9b00-e6d70dfdd872-kube-api-access-p5d46\" (UniqueName: \"kubernetes.io/projected/f916d575-cbfd-4e99-9b00-e6d70dfdd872-kube-api-access-p5d46\") pod \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") "
Oct 30 20:57:39.191969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.191824    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-model-cache\") pod \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") "
Oct 30 20:57:39.191969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.191854    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-dshm\") pod \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") "
Oct 30 20:57:39.192139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.191962    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tmp-dir\") pod \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") "
Oct 30 20:57:39.192139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192016    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tls-certs\") pod \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") "
Oct 30 20:57:39.192139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192110    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-model-cache\") pod \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") "
Oct 30 20:57:39.192341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192155    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-kube-api-access-46jcq\" (UniqueName: \"kubernetes.io/projected/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-kube-api-access-46jcq\") pod \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") "
Oct 30 20:57:39.192341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192158    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-model-cache" pod "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" (UID: "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.192341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192191    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tmp-dir" pod "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" (UID: "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.192341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192197    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tls-certs\" (UniqueName: \"kubernetes.io/secret/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tls-certs\") pod \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") "
Oct 30 20:57:39.192341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192229    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-home\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-home\") pod \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") "
Oct 30 20:57:39.192341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192259    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-dshm\") pod \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") "
Oct 30 20:57:39.192341 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192286    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-plugins-config-volume" pod "441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" (UID: "441e00a2-43ce-41ef-a1cb-0a5e32bd5c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:39.192777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192349    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-kube-api-access-bc5xb\" (UniqueName: \"kubernetes.io/projected/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-kube-api-access-bc5xb\") pod \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\" (UID: \"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19\") "
Oct 30 20:57:39.192777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192361    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-model-cache" pod "f916d575-cbfd-4e99-9b00-e6d70dfdd872" (UID: "f916d575-cbfd-4e99-9b00-e6d70dfdd872"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.192777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192455    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tmp-dir\") pod \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\" (UID: \"f916d575-cbfd-4e99-9b00-e6d70dfdd872\") "
Oct 30 20:57:39.192777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192493    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-home\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-home\") pod \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\" (UID: \"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a\") "
Oct 30 20:57:39.192777 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.192723    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-home" pod "f916d575-cbfd-4e99-9b00-e6d70dfdd872" (UID: "f916d575-cbfd-4e99-9b00-e6d70dfdd872"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193324    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193351    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193370    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9bgb\" (UniqueName: \"kubernetes.io/projected/46a5c650-1c2b-49da-b4f0-74ef63f02530-kube-api-access-t9bgb\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193386    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193400    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193415    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193566    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-home" pod "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" (UID: "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.193601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.193589    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tmp-dir" pod "f916d575-cbfd-4e99-9b00-e6d70dfdd872" (UID: "f916d575-cbfd-4e99-9b00-e6d70dfdd872"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.194676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.194655    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-kube-api-access-46jcq" pod "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" (UID: "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a"). InnerVolumeSpecName "kube-api-access-46jcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.195486 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.195462    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f916d575-cbfd-4e99-9b00-e6d70dfdd872-kube-api-access-p5d46" pod "f916d575-cbfd-4e99-9b00-e6d70dfdd872" (UID: "f916d575-cbfd-4e99-9b00-e6d70dfdd872"). InnerVolumeSpecName "kube-api-access-p5d46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.195616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.195495    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-kube-api-access-bc5xb" pod "441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" (UID: "441e00a2-43ce-41ef-a1cb-0a5e32bd5c19"). InnerVolumeSpecName "kube-api-access-bc5xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.195616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.195506    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-dshm" pod "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" (UID: "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.195616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.195509    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tls-certs" pod "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" (UID: "26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:39.195616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.195591    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tls-certs" pod "f916d575-cbfd-4e99-9b00-e6d70dfdd872" (UID: "f916d575-cbfd-4e99-9b00-e6d70dfdd872"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:39.196039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.196024    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-dshm" pod "f916d575-cbfd-4e99-9b00-e6d70dfdd872" (UID: "f916d575-cbfd-4e99-9b00-e6d70dfdd872"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.295056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295026    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5d46\" (UniqueName: \"kubernetes.io/projected/f916d575-cbfd-4e99-9b00-e6d70dfdd872-kube-api-access-p5d46\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295054    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295056 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295065    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295074    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46jcq\" (UniqueName: \"kubernetes.io/projected/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-kube-api-access-46jcq\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295084    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295103    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295111    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc5xb\" (UniqueName: \"kubernetes.io/projected/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19-kube-api-access-bc5xb\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295119    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f916d575-cbfd-4e99-9b00-e6d70dfdd872-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.295336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.295127    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.933265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.933185    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"
Oct 30 20:57:39.933265 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.933205    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm" event={"ID":"441e00a2-43ce-41ef-a1cb-0a5e32bd5c19","Type":"ContainerDied","Data":"00aa34c73cf4a44ece73960c424869e4ae73bc9d7ce60457b214caa4ca2189ab"}
Oct 30 20:57:39.933839 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.933272    5002 scope.go:122] "RemoveContainer" containerID="aa94307b74bc3a349afac548bf08a1b475fa8681516de38c1af45716f428e1b3"
Oct 30 20:57:39.935028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.934964    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h" event={"ID":"46a5c650-1c2b-49da-b4f0-74ef63f02530","Type":"ContainerDied","Data":"6291d3977d436e979457db14ae0dd6d17fa0d8b2ddab98cb5d3cbeb8e5f5a54e"}
Oct 30 20:57:39.935103 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.935033    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"
Oct 30 20:57:39.954288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.954269    5002 scope.go:122] "RemoveContainer" containerID="b379cdb9b66412cfca094a4f5f72b438c4e59d26de06d4b961b0db577238e534"
Oct 30 20:57:39.954694 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.954670    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8" event={"ID":"f916d575-cbfd-4e99-9b00-e6d70dfdd872","Type":"ContainerDied","Data":"9a9275343f433e148c8169c21658e05a321897e28acbf66b398640e2270687a3"}
Oct 30 20:57:39.954803 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.954733    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"
Oct 30 20:57:39.957410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.957387    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz" event={"ID":"26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a","Type":"ContainerDied","Data":"e148639e328f9c1e8e01b2eab8bce179fef2ae07c0c9e4d9fcadaff68413314f"}
Oct 30 20:57:39.957516 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.957459    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"
Oct 30 20:57:39.966194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.966163    5002 scope.go:122] "RemoveContainer" containerID="b1864d37532fddd56be5d74fc98e867e9d58a0434c1f7aec28642c8e5a3bbdb0"
Oct 30 20:57:39.975980 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:39.975961    5002 scope.go:122] "RemoveContainer" containerID="9e8150813c2419f6712694f013b0c07cdaf2b38142ce101a338441d4b00e7689"
Oct 30 20:57:40.102701 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.102674    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"]
Oct 30 20:57:40.152701 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.152682    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-w5-a78bf4-kserve-7d47585f44-tjbqz"]
Oct 30 20:57:40.216700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.216675    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"]
Oct 30 20:57:40.220920 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.220901    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm"]
Oct 30 20:57:40.332139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.332115    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"]
Oct 30 20:57:40.336924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.336903    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h"]
Oct 30 20:57:40.442261 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.442235    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"]
Oct 30 20:57:40.449363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.449341    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/distinct-2-w5-a78bf4-kserve-657bcbdc95-h87s8"]
Oct 30 20:57:40.794372 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:40.794320    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:41.105337 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:41.105243    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" podUID="81a9e296-a407-4ef0-878b-5c10dea1613f" containerName="payload-processing" containerID="cri-o://37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7" gracePeriod=25
Oct 30 20:57:41.217371 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:41.217300    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" podUID="0d0423fc-58b9-42f3-941c-f664d7646730" containerName="payload-pre-processing" containerID="cri-o://290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e" gracePeriod=25
Oct 30 20:57:41.968269 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:41.968237    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" path="/var/lib/kubelet/pods/26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a/volumes"
Oct 30 20:57:41.968665 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:41.968652    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" path="/var/lib/kubelet/pods/441e00a2-43ce-41ef-a1cb-0a5e32bd5c19/volumes"
Oct 30 20:57:41.969013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:41.969000    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a5c650-1c2b-49da-b4f0-74ef63f02530" path="/var/lib/kubelet/pods/46a5c650-1c2b-49da-b4f0-74ef63f02530/volumes"
Oct 30 20:57:41.969749 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:41.969729    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" path="/var/lib/kubelet/pods/f916d575-cbfd-4e99-9b00-e6d70dfdd872/volumes"
Oct 30 20:57:42.421378 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:42.421265    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:42.600872 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:42.600844    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:43.239627 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:43.239588    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:43.502184 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:43.502071    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:43.674858 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:43.674830    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:45.509464 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:45.509377    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:46.442738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:46.442701    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:57:48.586327 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:48.586287    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" podUID="81a9e296-a407-4ef0-878b-5c10dea1613f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:48.586720 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:48.586302    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" podUID="0d0423fc-58b9-42f3-941c-f664d7646730" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:49.051303 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:49.051268    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:49.248990 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:49.248958    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:50.307714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:50.307673    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:50.744340 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:50.744304    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:51.030144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:51.030055    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:51.983628 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:51.983592    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:52.168825 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:52.168782    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:55.645273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:55.645221    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:55.871499 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:55.871461    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:56.072217 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:56.072174    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:56.258491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:56.258461    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:57.116704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.116672    5002 generic.go:356] "Generic (PLEG): container finished" podID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerID="cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4" exitCode=0
Oct 30 20:57:57.117103 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.116743    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" event={"ID":"2a4d728a-78c0-405c-9907-6df33d6bd997","Type":"ContainerDied","Data":"cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4"}
Oct 30 20:57:57.117170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.117151    5002 scope.go:122] "RemoveContainer" containerID="cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4"
Oct 30 20:57:57.118701 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.118666    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:57.118813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.118740    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:57.118813 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.118770    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:57.228934 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.228901    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:57.418266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:57.418164    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:58.146835 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:58.146798    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" event={"ID":"2a4d728a-78c0-405c-9907-6df33d6bd997","Type":"ContainerStarted","Data":"640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406"}
Oct 30 20:57:58.586028 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:58.585992    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" podUID="0d0423fc-58b9-42f3-941c-f664d7646730" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:58.586277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:58.585998    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" podUID="81a9e296-a407-4ef0-878b-5c10dea1613f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:58.796297 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:58.796267    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:57:58.797298 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:58.797267    5002 patch_prober.go:29] interesting pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:57:58.797461 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:58.797322    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:57:59.133557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.119306    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:59.332926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.332881    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:59.480294 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480264    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"]
Oct 30 20:57:59.480880 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480866    5002 cpu_manager.go:414] "containerMap: removing container" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480882    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480900    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480904    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480918    5002 cpu_manager.go:414] "containerMap: removing container" podUID="441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" containerName="payload-pre-processing"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480923    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" containerName="payload-pre-processing"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480931    5002 cpu_manager.go:414] "containerMap: removing container" podUID="46a5c650-1c2b-49da-b4f0-74ef63f02530" containerName="payload-processing"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480936    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a5c650-1c2b-49da-b4f0-74ef63f02530" containerName="payload-processing"
Oct 30 20:57:59.480944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480946    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy"
Oct 30 20:57:59.481230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.480951    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy"
Oct 30 20:57:59.481230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.481060    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="26bd6dcf-4485-4d9c-8bd3-02e8f1f0af4a" containerName="main"
Oct 30 20:57:59.481230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.481067    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f916d575-cbfd-4e99-9b00-e6d70dfdd872" containerName="main"
Oct 30 20:57:59.481230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.481073    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="441e00a2-43ce-41ef-a1cb-0a5e32bd5c19" containerName="payload-pre-processing"
Oct 30 20:57:59.481230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.481078    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a5c650-1c2b-49da-b4f0-74ef63f02530" containerName="payload-processing"
Oct 30 20:57:59.481230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.481086    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cffdbee5-8fe7-4e9f-8eb2-4ebe76507b71" containerName="istio-proxy"
Oct 30 20:57:59.484764 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.484747    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.541006 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.540978    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-23491e\""
Oct 30 20:57:59.541190 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.540987    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-23491e-dockercfg-x45zf\""
Oct 30 20:57:59.546394 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.546372    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"]
Oct 30 20:57:59.581367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.581326    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.581756 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.581734    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dksdj\" (UniqueName: \"kubernetes.io/projected/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-kube-api-access-dksdj\") pod \"payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.670513 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.670475    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"]
Oct 30 20:57:59.676172 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.676150    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:57:59.682405 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.682382    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4298da21-8c66-4510-af57-9f4efc8fce69-plugins-config-volume\") pod \"payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") " pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:57:59.682560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.682457    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.682560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.682541    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dksdj\" (UniqueName: \"kubernetes.io/projected/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-kube-api-access-dksdj\") pod \"payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.682703 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.682604    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gv9\" (UniqueName: \"kubernetes.io/projected/4298da21-8c66-4510-af57-9f4efc8fce69-kube-api-access-d8gv9\") pod \"payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") " pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:57:59.683326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.683304    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.694088 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.693454    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"]
Oct 30 20:57:59.749322 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.749249    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dksdj\" (UniqueName: \"kubernetes.io/projected/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-kube-api-access-dksdj\") pod \"payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.783704 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.783676    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gv9\" (UniqueName: \"kubernetes.io/projected/4298da21-8c66-4510-af57-9f4efc8fce69-kube-api-access-d8gv9\") pod \"payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") " pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:57:59.783867 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.783801    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4298da21-8c66-4510-af57-9f4efc8fce69-plugins-config-volume\") pod \"payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") " pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:57:59.784421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.784401    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4298da21-8c66-4510-af57-9f4efc8fce69-plugins-config-volume\") pod \"payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") " pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:57:59.796550 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.796499    5002 patch_prober.go:29] interesting pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:57:59.796672 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.796577    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:57:59.798333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.798310    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:57:59.845827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.845803    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gv9\" (UniqueName: \"kubernetes.io/projected/4298da21-8c66-4510-af57-9f4efc8fce69-kube-api-access-d8gv9\") pod \"payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") " pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:57:59.990282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:57:59.990055    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:58:00.168926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:00.168816    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:00.250759 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:00.250729    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"]
Oct 30 20:58:00.253455 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:58:00.252925    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5c2f4b_d17a_4d11_8ad8_f5b91a17de3d.slice/crio-132f2c0f500a4b3e0dafd3c377fead9315c5917c4c7b47fb406c7566d8eccc72 WatchSource:0}: Error finding container 132f2c0f500a4b3e0dafd3c377fead9315c5917c4c7b47fb406c7566d8eccc72: Status 404 returned error can't find the container with id 132f2c0f500a4b3e0dafd3c377fead9315c5917c4c7b47fb406c7566d8eccc72
Oct 30 20:58:00.350821 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:00.350279    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:00.450513 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:00.450480    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"]
Oct 30 20:58:00.796725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:00.796651    5002 patch_prober.go:29] interesting pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:58:00.796725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:00.796701    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:58:00.877496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:00.877453    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:01.183234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.183200    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx" event={"ID":"4298da21-8c66-4510-af57-9f4efc8fce69","Type":"ContainerStarted","Data":"83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed"}
Oct 30 20:58:01.183398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.183300    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx" event={"ID":"4298da21-8c66-4510-af57-9f4efc8fce69","Type":"ContainerStarted","Data":"b6140347b38a93d16d0e660c332e3a643a967d083023f07976b9a4075a34470d"}
Oct 30 20:58:01.183398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.183332    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:58:01.184731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.184701    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf" event={"ID":"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d","Type":"ContainerStarted","Data":"747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb"}
Oct 30 20:58:01.184868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.184751    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf" event={"ID":"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d","Type":"ContainerStarted","Data":"132f2c0f500a4b3e0dafd3c377fead9315c5917c4c7b47fb406c7566d8eccc72"}
Oct 30 20:58:01.184868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.184809    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:58:01.359973 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.358963    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx" podStartSLOduration=2.358948018 podStartE2EDuration="2.358948018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:01.312956691 +0000 UTC m=+2307.462007679" watchObservedRunningTime="2026-10-30 20:58:01.358948018 +0000 UTC m=+2307.507999005"
Oct 30 20:58:01.501325 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.498163    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf" podStartSLOduration=2.4981431929999998 podStartE2EDuration="2.498143193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:01.432000801 +0000 UTC m=+2307.581051789" watchObservedRunningTime="2026-10-30 20:58:01.498143193 +0000 UTC m=+2307.647194181"
Oct 30 20:58:01.572223 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.571771    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:58:01.574243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.574225    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:58:01.608738 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.608713    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a9e296-a407-4ef0-878b-5c10dea1613f-kube-api-access-97xnb\" (UniqueName: \"kubernetes.io/projected/81a9e296-a407-4ef0-878b-5c10dea1613f-kube-api-access-97xnb\") pod \"81a9e296-a407-4ef0-878b-5c10dea1613f\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") "
Oct 30 20:58:01.608870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.608756    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0423fc-58b9-42f3-941c-f664d7646730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0423fc-58b9-42f3-941c-f664d7646730-plugins-config-volume\") pod \"0d0423fc-58b9-42f3-941c-f664d7646730\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") "
Oct 30 20:58:01.608870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.608807    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a9e296-a407-4ef0-878b-5c10dea1613f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a9e296-a407-4ef0-878b-5c10dea1613f-plugins-config-volume\") pod \"81a9e296-a407-4ef0-878b-5c10dea1613f\" (UID: \"81a9e296-a407-4ef0-878b-5c10dea1613f\") "
Oct 30 20:58:01.608870 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.608853    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0423fc-58b9-42f3-941c-f664d7646730-kube-api-access-kcbfm\" (UniqueName: \"kubernetes.io/projected/0d0423fc-58b9-42f3-941c-f664d7646730-kube-api-access-kcbfm\") pod \"0d0423fc-58b9-42f3-941c-f664d7646730\" (UID: \"0d0423fc-58b9-42f3-941c-f664d7646730\") "
Oct 30 20:58:01.609170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.609145    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0423fc-58b9-42f3-941c-f664d7646730-plugins-config-volume" pod "0d0423fc-58b9-42f3-941c-f664d7646730" (UID: "0d0423fc-58b9-42f3-941c-f664d7646730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:01.609263 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.609249    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0423fc-58b9-42f3-941c-f664d7646730-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:01.609323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.609288    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a9e296-a407-4ef0-878b-5c10dea1613f-plugins-config-volume" pod "81a9e296-a407-4ef0-878b-5c10dea1613f" (UID: "81a9e296-a407-4ef0-878b-5c10dea1613f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:01.646897 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.646843    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a9e296-a407-4ef0-878b-5c10dea1613f-kube-api-access-97xnb" pod "81a9e296-a407-4ef0-878b-5c10dea1613f" (UID: "81a9e296-a407-4ef0-878b-5c10dea1613f"). InnerVolumeSpecName "kube-api-access-97xnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:01.647058 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.646996    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0423fc-58b9-42f3-941c-f664d7646730-kube-api-access-kcbfm" pod "0d0423fc-58b9-42f3-941c-f664d7646730" (UID: "0d0423fc-58b9-42f3-941c-f664d7646730"). InnerVolumeSpecName "kube-api-access-kcbfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:01.704141 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.697326    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:01.749315 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.749254    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97xnb\" (UniqueName: \"kubernetes.io/projected/81a9e296-a407-4ef0-878b-5c10dea1613f-kube-api-access-97xnb\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:01.749650 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.749632    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a9e296-a407-4ef0-878b-5c10dea1613f-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:01.749789 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.749762    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcbfm\" (UniqueName: \"kubernetes.io/projected/0d0423fc-58b9-42f3-941c-f664d7646730-kube-api-access-kcbfm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:01.796731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.796696    5002 patch_prober.go:29] interesting pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:58:01.796850 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:01.796754    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:58:02.193966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.193930    5002 generic.go:356] "Generic (PLEG): container finished" podID="0d0423fc-58b9-42f3-941c-f664d7646730" containerID="290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e" exitCode=0
Oct 30 20:58:02.194170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.193983    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"
Oct 30 20:58:02.194170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.193992    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" event={"ID":"0d0423fc-58b9-42f3-941c-f664d7646730","Type":"ContainerDied","Data":"290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e"}
Oct 30 20:58:02.194170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.194051    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr" event={"ID":"0d0423fc-58b9-42f3-941c-f664d7646730","Type":"ContainerDied","Data":"5403f309efb2d49cbfe83aac009857a5bc8c5bb7f456e22bb878a346bc3bcb92"}
Oct 30 20:58:02.194170 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.194079    5002 scope.go:122] "RemoveContainer" containerID="290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e"
Oct 30 20:58:02.195752 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.195727    5002 generic.go:356] "Generic (PLEG): container finished" podID="81a9e296-a407-4ef0-878b-5c10dea1613f" containerID="37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7" exitCode=0
Oct 30 20:58:02.196349 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.196330    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" event={"ID":"81a9e296-a407-4ef0-878b-5c10dea1613f","Type":"ContainerDied","Data":"37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7"}
Oct 30 20:58:02.196449 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.196382    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw" event={"ID":"81a9e296-a407-4ef0-878b-5c10dea1613f","Type":"ContainerDied","Data":"16a18990ce1703771285ac82c8a830d3f8433b3577bf5009dc517940e75badc4"}
Oct 30 20:58:02.196519 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.196452    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"
Oct 30 20:58:02.206379 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.206352    5002 scope.go:122] "RemoveContainer" containerID="290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e"
Oct 30 20:58:02.206681 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:02.206648    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e\": container with ID starting with 290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e not found: ID does not exist" containerID="290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e"
Oct 30 20:58:02.206801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.206687    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e"} err="failed to get container status \"290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e\": rpc error: code = NotFound desc = could not find container \"290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e\": container with ID starting with 290b1c85b39022ab16a37aa8e2eb515c99f3c51e3a1d3ebb3dcfba9aae9d400e not found: ID does not exist"
Oct 30 20:58:02.206801 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.206703    5002 scope.go:122] "RemoveContainer" containerID="37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7"
Oct 30 20:58:02.249384 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.249277    5002 scope.go:122] "RemoveContainer" containerID="37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7"
Oct 30 20:58:02.249736 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:02.249672    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7\": container with ID starting with 37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7 not found: ID does not exist" containerID="37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7"
Oct 30 20:58:02.249736 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.249712    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7"} err="failed to get container status \"37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7\": rpc error: code = NotFound desc = could not find container \"37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7\": container with ID starting with 37f6ecb7684d4dbd4a5fc6fbf8f71b5d0dced844eedcfc02d1732d61ac7cefb7 not found: ID does not exist"
Oct 30 20:58:02.363583 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.363506    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"]
Oct 30 20:58:02.435564 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.433519    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw"]
Oct 30 20:58:02.480370 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.478629    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"]
Oct 30 20:58:02.484574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.484062    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr"]
Oct 30 20:58:02.796441 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.796406    5002 patch_prober.go:29] interesting pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:58:02.796639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:02.796468    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:58:03.119112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:03.118660    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:58:03.800468 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:03.800441    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:58:03.801427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:03.801409    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 20:58:03.966495 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:03.966460    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0423fc-58b9-42f3-941c-f664d7646730" path="/var/lib/kubelet/pods/0d0423fc-58b9-42f3-941c-f664d7646730/volumes"
Oct 30 20:58:03.966863 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:03.966849    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a9e296-a407-4ef0-878b-5c10dea1613f" path="/var/lib/kubelet/pods/81a9e296-a407-4ef0-878b-5c10dea1613f/volumes"
Oct 30 20:58:05.129725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:05.129695    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:05.727155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:05.727123    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:07.115035 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:07.114993    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:58:07.854582 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:07.854554    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:08.102267 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:08.102231    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:08.936755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:08.935650    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:10.151073 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.151042    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"]
Oct 30 20:58:10.151649 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.151634    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0d0423fc-58b9-42f3-941c-f664d7646730" containerName="payload-pre-processing"
Oct 30 20:58:10.151649 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.151649    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0423fc-58b9-42f3-941c-f664d7646730" containerName="payload-pre-processing"
Oct 30 20:58:10.151755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.151681    5002 cpu_manager.go:414] "containerMap: removing container" podUID="81a9e296-a407-4ef0-878b-5c10dea1613f" containerName="payload-processing"
Oct 30 20:58:10.151755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.151687    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a9e296-a407-4ef0-878b-5c10dea1613f" containerName="payload-processing"
Oct 30 20:58:10.151819 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.151781    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0423fc-58b9-42f3-941c-f664d7646730" containerName="payload-pre-processing"
Oct 30 20:58:10.151819 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.151795    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a9e296-a407-4ef0-878b-5c10dea1613f" containerName="payload-processing"
Oct 30 20:58:10.156731 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.156716    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.212745 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.212723    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"]
Oct 30 20:58:10.249836 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.249804    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghvv\" (UniqueName: \"kubernetes.io/projected/e430c6d2-88ee-413f-8b7d-f90efe62a92f-kube-api-access-2ghvv\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.249995 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.249893    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430c6d2-88ee-413f-8b7d-f90efe62a92f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.250072 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.250014    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.250072 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.250061    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-metrics-certs\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.259149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.259125    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-11a0b0-serving-cert\""
Oct 30 20:58:10.351654 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.351625    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghvv\" (UniqueName: \"kubernetes.io/projected/e430c6d2-88ee-413f-8b7d-f90efe62a92f-kube-api-access-2ghvv\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.351795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.351684    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430c6d2-88ee-413f-8b7d-f90efe62a92f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.351795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.351751    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.351901 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:10.351876    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-serving-cert: secret "maas-api-e2e-inf-b-11a0b0-serving-cert" not found
Oct 30 20:58:10.351961 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.351890    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-metrics-certs\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.352014 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:10.351960    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls podName:e430c6d2-88ee-413f-8b7d-f90efe62a92f nodeName:}" failed. No retries permitted until 2026-10-30 20:58:10.851934595 +0000 UTC m=+2317.000985562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls") pod "maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq" (UID: "e430c6d2-88ee-413f-8b7d-f90efe62a92f") : secret "maas-api-e2e-inf-b-11a0b0-serving-cert" not found
Oct 30 20:58:10.352410 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.352391    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430c6d2-88ee-413f-8b7d-f90efe62a92f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.354344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.354321    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-metrics-certs\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.379451 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.379430    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:10.413448 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.413369    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghvv\" (UniqueName: \"kubernetes.io/projected/e430c6d2-88ee-413f-8b7d-f90efe62a92f-kube-api-access-2ghvv\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.566968 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.566932    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:10.740416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.740383    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:10.859269 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.859241    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:10.861686 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:10.861661    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls\") pod \"maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:11.069511 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:11.069436    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:11.446546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:11.446504    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"]
Oct 30 20:58:11.447126 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:58:11.447086    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode430c6d2_88ee_413f_8b7d_f90efe62a92f.slice/crio-63a4504b7e4ad609c808fef8ef167f95cc35ca95a83fcefa26df1d02b5e23ff9 WatchSource:0}: Error finding container 63a4504b7e4ad609c808fef8ef167f95cc35ca95a83fcefa26df1d02b5e23ff9: Status 404 returned error can't find the container with id 63a4504b7e4ad609c808fef8ef167f95cc35ca95a83fcefa26df1d02b5e23ff9
Oct 30 20:58:12.200275 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:12.200238    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 20:58:12.200481 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:12.200376    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 20:58:12.276928 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:12.276865    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq" event={"ID":"e430c6d2-88ee-413f-8b7d-f90efe62a92f","Type":"ContainerStarted","Data":"63a4504b7e4ad609c808fef8ef167f95cc35ca95a83fcefa26df1d02b5e23ff9"}
Oct 30 20:58:13.287554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:13.287504    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq" event={"ID":"e430c6d2-88ee-413f-8b7d-f90efe62a92f","Type":"ContainerStarted","Data":"4cffe95c08b985318b7cc8865efb0296882a70badfd96ace8cb6e36fd9dfbed5"}
Oct 30 20:58:13.288048 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:13.287581    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:13.408192 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:13.408141    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq" podStartSLOduration=2.553996998 podStartE2EDuration="3.408124285s" totalImagesPullingTime="854.127287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:13.405161126 +0000 UTC m=+2319.554212112" watchObservedRunningTime="2026-10-30 20:58:13.408124285 +0000 UTC m=+2319.557175280"
Oct 30 20:58:14.622271 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:14.622229    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:15.321266 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:15.321225    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:19.299647 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:19.299614    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 20:58:20.202230 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:20.202189    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:22.333852 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:22.333817    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:24.376276 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:24.376239    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:58:24.986993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:24.986955    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:27.941575 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:27.930578    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:28.655842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:28.655802    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:29.141376 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.141340    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"]
Oct 30 20:58:29.145723 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.145704    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191390    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzjt\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-kube-api-access-4wzjt\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191479    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-token\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191548    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-socket\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191589    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-envoy\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191647    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-credential-socket\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191673    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-podinfo\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191728    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-data\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191763    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-certs\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.193558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.191899    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istiod-ca-cert\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.205924 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.205888    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-92f229-openshift-default-dockercfg-fzwlt\""
Oct 30 20:58:29.262648 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.262617    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"]
Oct 30 20:58:29.275130 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.275076    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:29.292639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.292613    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istiod-ca-cert\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.292919 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.292883    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzjt\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-kube-api-access-4wzjt\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.292966    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-token\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293014 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.292996    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-socket\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.293031    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-envoy\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.293079    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-credential-socket\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.293104    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-podinfo\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.293142    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-data\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293279 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.293178    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-certs\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.293573    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-certs\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.293889 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.293674    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istiod-ca-cert\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.294139 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.294117    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-credential-socket\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.295112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.294728    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-data\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.295112 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.294806    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-socket\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.303450 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.303422    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-envoy\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.304933 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.304908    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-podinfo\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.352661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.352593    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-token\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.353328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.353303    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzjt\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-kube-api-access-4wzjt\") pod \"e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") " pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.461913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.461882    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:29.645658 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.644623    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:29.945373 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.945333    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"]
Oct 30 20:58:29.947419 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:58:29.947388    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe31d77_e37f_41b1_87fb_8a3eb5374f40.slice/crio-4417e600659d1e22074149f9e19442bd4ff62e21d005ab0f0301d18d23070f9e WatchSource:0}: Error finding container 4417e600659d1e22074149f9e19442bd4ff62e21d005ab0f0301d18d23070f9e: Status 404 returned error can't find the container with id 4417e600659d1e22074149f9e19442bd4ff62e21d005ab0f0301d18d23070f9e
Oct 30 20:58:29.950242 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.950204    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:29.950352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.950287    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:29.950352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:29.950330    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:30.145558 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:30.144563    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:30.423504 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:30.423465    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" event={"ID":"efe31d77-e37f-41b1-87fb-8a3eb5374f40","Type":"ContainerStarted","Data":"70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a"}
Oct 30 20:58:30.423709 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:30.423548    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" event={"ID":"efe31d77-e37f-41b1-87fb-8a3eb5374f40","Type":"ContainerStarted","Data":"4417e600659d1e22074149f9e19442bd4ff62e21d005ab0f0301d18d23070f9e"}
Oct 30 20:58:30.462570 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:30.462517    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:30.545289 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:30.545226    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podStartSLOduration=1.5452094779999999 podStartE2EDuration="1.545209478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:30.542523247 +0000 UTC m=+2336.691574246" watchObservedRunningTime="2026-10-30 20:58:30.545209478 +0000 UTC m=+2336.694260466"
Oct 30 20:58:31.469054 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:31.469020    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:32.446969 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:32.446937    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:32.447925 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:32.447899    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 20:58:33.378748 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:33.378708    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:33.581849 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:33.581818    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:33.781952 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:33.781888    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:34.326008 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.325970    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"]
Oct 30 20:58:34.326232 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.326206    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s" podUID="f5002a04-b76d-44cf-8e5f-7063871fe6e1" containerName="maas-api" containerID="cri-o://ec46789e02779f176c14defd1d1fa178fa6f7a14ef082b70d0c82ee9386675d7" gracePeriod=30
Oct 30 20:58:34.466105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.466062    5002 generic.go:356] "Generic (PLEG): container finished" podID="f5002a04-b76d-44cf-8e5f-7063871fe6e1" containerID="ec46789e02779f176c14defd1d1fa178fa6f7a14ef082b70d0c82ee9386675d7" exitCode=0
Oct 30 20:58:34.466491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.466141    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s" event={"ID":"f5002a04-b76d-44cf-8e5f-7063871fe6e1","Type":"ContainerDied","Data":"ec46789e02779f176c14defd1d1fa178fa6f7a14ef082b70d0c82ee9386675d7"}
Oct 30 20:58:34.468209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.468184    5002 generic.go:356] "Generic (PLEG): container finished" podID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerID="e45614f34112a5c67e2aec0fb098cc08e6ea342cbb92aa2eceaf9eedc2e9cce3" exitCode=0
Oct 30 20:58:34.468336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.468263    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" event={"ID":"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f","Type":"ContainerDied","Data":"e45614f34112a5c67e2aec0fb098cc08e6ea342cbb92aa2eceaf9eedc2e9cce3"}
Oct 30 20:58:34.468790 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.468769    5002 scope.go:122] "RemoveContainer" containerID="e45614f34112a5c67e2aec0fb098cc08e6ea342cbb92aa2eceaf9eedc2e9cce3"
Oct 30 20:58:34.471755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.470269    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:34.471755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.470351    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:34.471755 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.470396    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:34.505326 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.505300    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:34.655645 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.655622    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:58:34.692333 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.692308    5002 scope.go:122] "RemoveContainer" containerID="ec46789e02779f176c14defd1d1fa178fa6f7a14ef082b70d0c82ee9386675d7"
Oct 30 20:58:34.768875 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.768842    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5002a04-b76d-44cf-8e5f-7063871fe6e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5002a04-b76d-44cf-8e5f-7063871fe6e1-openshift-service-ca\") pod \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") "
Oct 30 20:58:34.769273 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.769169    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5002a04-b76d-44cf-8e5f-7063871fe6e1-openshift-service-ca" pod "f5002a04-b76d-44cf-8e5f-7063871fe6e1" (UID: "f5002a04-b76d-44cf-8e5f-7063871fe6e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:34.769422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.769404    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls\") pod \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") "
Oct 30 20:58:34.769829 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.769807    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5002a04-b76d-44cf-8e5f-7063871fe6e1-kube-api-access-j7t6f\" (UniqueName: \"kubernetes.io/projected/f5002a04-b76d-44cf-8e5f-7063871fe6e1-kube-api-access-j7t6f\") pod \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") "
Oct 30 20:58:34.770176 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.770156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-metrics-certs\") pod \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\" (UID: \"f5002a04-b76d-44cf-8e5f-7063871fe6e1\") "
Oct 30 20:58:34.771186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.771130    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5002a04-b76d-44cf-8e5f-7063871fe6e1-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:34.781869 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.780612    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-metrics-certs" pod "f5002a04-b76d-44cf-8e5f-7063871fe6e1" (UID: "f5002a04-b76d-44cf-8e5f-7063871fe6e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:34.791730 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.791698    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls" pod "f5002a04-b76d-44cf-8e5f-7063871fe6e1" (UID: "f5002a04-b76d-44cf-8e5f-7063871fe6e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:34.807746 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.807713    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5002a04-b76d-44cf-8e5f-7063871fe6e1-kube-api-access-j7t6f" pod "f5002a04-b76d-44cf-8e5f-7063871fe6e1" (UID: "f5002a04-b76d-44cf-8e5f-7063871fe6e1"). InnerVolumeSpecName "kube-api-access-j7t6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:34.872913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.872878    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7t6f\" (UniqueName: \"kubernetes.io/projected/f5002a04-b76d-44cf-8e5f-7063871fe6e1-kube-api-access-j7t6f\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:34.872913 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.872911    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:34.873142 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:34.872923    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5002a04-b76d-44cf-8e5f-7063871fe6e1-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:35.484514 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.484472    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s" event={"ID":"f5002a04-b76d-44cf-8e5f-7063871fe6e1","Type":"ContainerDied","Data":"4a15ac75c4ded9aa1123dd0306c49cd6cc629feb99b351ddbe017cc37640ae17"}
Oct 30 20:58:35.487328 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.487306    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"
Oct 30 20:58:35.487656 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.487628    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" event={"ID":"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f","Type":"ContainerStarted","Data":"6697fcc4065760059cc61045a3e7382a777aecebd75861f2e324918656b63b2d"}
Oct 30 20:58:35.743612 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.742980    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"]
Oct 30 20:58:35.746669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.746631    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f9b95-75c9ffddcd-t6s7s"]
Oct 30 20:58:35.770715 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.770693    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:58:35.771588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.771526    5002 patch_prober.go:29] interesting pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:58:35.771645 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.771618    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:58:35.966799 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:35.966762    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5002a04-b76d-44cf-8e5f-7063871fe6e1" path="/var/lib/kubelet/pods/f5002a04-b76d-44cf-8e5f-7063871fe6e1/volumes"
Oct 30 20:58:36.328562 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:36.327948    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:36.504443 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:36.504415    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:36.677119 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:36.677028    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:36.774560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:36.773493    5002 patch_prober.go:29] interesting pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:58:36.774560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:36.773666    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:58:36.985596 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:36.983016    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:37.146483 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:37.146446    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:58:37.174935 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.174909    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:37.351674 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.351598    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"]
Oct 30 20:58:37.352563 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.352523    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f5002a04-b76d-44cf-8e5f-7063871fe6e1" containerName="maas-api"
Oct 30 20:58:37.352668 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.352564    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5002a04-b76d-44cf-8e5f-7063871fe6e1" containerName="maas-api"
Oct 30 20:58:37.352760 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.352748    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5002a04-b76d-44cf-8e5f-7063871fe6e1" containerName="maas-api"
Oct 30 20:58:37.358522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.358495    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"]
Oct 30 20:58:37.358685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.358632    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.365187 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.364070    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:37.408611 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.408584    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"]
Oct 30 20:58:37.427560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.426884    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-92f229-dockercfg-l9jnl\""
Oct 30 20:58:37.427560 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.427034    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-92f229\""
Oct 30 20:58:37.446223 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.446178    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16c027d-c495-4c3d-b5c5-b0a51016d26d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") " pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.446338 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.446241    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252kz\" (UniqueName: \"kubernetes.io/projected/e16c027d-c495-4c3d-b5c5-b0a51016d26d-kube-api-access-252kz\") pod \"payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") " pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.448422 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.448401    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.469442 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.469420    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"]
Oct 30 20:58:37.547491 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.547451    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946fh\" (UniqueName: \"kubernetes.io/projected/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-kube-api-access-946fh\") pod \"payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.548000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.547619    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16c027d-c495-4c3d-b5c5-b0a51016d26d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") " pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.548000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.547658    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252kz\" (UniqueName: \"kubernetes.io/projected/e16c027d-c495-4c3d-b5c5-b0a51016d26d-kube-api-access-252kz\") pod \"payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") " pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.548000 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.547780    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.548730 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.548706    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16c027d-c495-4c3d-b5c5-b0a51016d26d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") " pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.608291 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.608226    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252kz\" (UniqueName: \"kubernetes.io/projected/e16c027d-c495-4c3d-b5c5-b0a51016d26d-kube-api-access-252kz\") pod \"payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") " pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.649039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.649000    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.649194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.649166    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946fh\" (UniqueName: \"kubernetes.io/projected/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-kube-api-access-946fh\") pod \"payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.649809 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.649787    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.676194 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.676172    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:37.706979 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.706958    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946fh\" (UniqueName: \"kubernetes.io/projected/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-kube-api-access-946fh\") pod \"payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.768103 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.767814    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:37.770944 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.770903    5002 patch_prober.go:29] interesting pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:58:37.771051 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:37.770957    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:58:38.090676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.090638    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"]
Oct 30 20:58:38.207383 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.207357    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"]
Oct 30 20:58:38.208474 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:58:38.208449    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0ffbde_0875_4393_8f9b_2d4d4b9d9c5b.slice/crio-65e3db0193c491403276cf919f8270a6c7d958e3243a3a8da5f09f38a295670b WatchSource:0}: Error finding container 65e3db0193c491403276cf919f8270a6c7d958e3243a3a8da5f09f38a295670b: Status 404 returned error can't find the container with id 65e3db0193c491403276cf919f8270a6c7d958e3243a3a8da5f09f38a295670b
Oct 30 20:58:38.509360 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.509327    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67" event={"ID":"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b","Type":"ContainerStarted","Data":"ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b"}
Oct 30 20:58:38.509567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.509384    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67" event={"ID":"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b","Type":"ContainerStarted","Data":"65e3db0193c491403276cf919f8270a6c7d958e3243a3a8da5f09f38a295670b"}
Oct 30 20:58:38.509567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.509409    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:38.532556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.517430    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8" event={"ID":"e16c027d-c495-4c3d-b5c5-b0a51016d26d","Type":"ContainerStarted","Data":"6da7a489c51259305aee47787f7e43e808a40d4502aa9b459b51cb75479c7b49"}
Oct 30 20:58:38.532556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.517494    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8" event={"ID":"e16c027d-c495-4c3d-b5c5-b0a51016d26d","Type":"ContainerStarted","Data":"aab68a2df550782a2b6f223972723d034e9ac4d87a6879e4931828edf25be93d"}
Oct 30 20:58:38.532556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.518174    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:38.646591 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.646192    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67" podStartSLOduration=1.6461736249999999 podStartE2EDuration="1.646173625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:38.64275424 +0000 UTC m=+2344.791805229" watchObservedRunningTime="2026-10-30 20:58:38.646173625 +0000 UTC m=+2344.795224612"
Oct 30 20:58:38.763546 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.762364    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8" podStartSLOduration=1.762346012 podStartE2EDuration="1.762346012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:38.761025896 +0000 UTC m=+2344.910076890" watchObservedRunningTime="2026-10-30 20:58:38.762346012 +0000 UTC m=+2344.911397028"
Oct 30 20:58:38.771015 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.770986    5002 patch_prober.go:29] interesting pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:58:38.771164 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:38.771033    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:58:39.661630 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:39.661600    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:58:39.771234 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:39.771205    5002 patch_prober.go:29] interesting pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:58:39.771427 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:39.771250    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:58:40.887613 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:40.887579    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:58:40.889859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:40.889834    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 20:58:41.553272 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:41.553240    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:44.111389 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:44.111357    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:44.288661 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:44.288623    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:44.409804 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:44.409730    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"]
Oct 30 20:58:44.409972 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:44.409924    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" containerID="cri-o://9cc42d1ebdac21aa5df70ebdfc627bccb6b4c5b128aec6beb147740a91ca881f" gracePeriod=30
Oct 30 20:58:44.499553 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:44.499359    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:45.293842 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:45.293810    5002 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:58:45.294229 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:45.293861    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:58:46.550840 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:46.550804    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:46.958975 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:46.958888    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:47.148269 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:47.148238    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:47.357921 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:47.357829    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:48.660508 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:48.660477    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:48.844881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:48.844849    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:49.548390 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.547696    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 20:58:49.624850 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.624818    5002 generic.go:356] "Generic (PLEG): container finished" podID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerID="9cc42d1ebdac21aa5df70ebdfc627bccb6b4c5b128aec6beb147740a91ca881f" exitCode=0
Oct 30 20:58:49.625039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.624886    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" event={"ID":"e6ad1cb3-708e-48f5-abe2-be61ea83e237","Type":"ContainerDied","Data":"9cc42d1ebdac21aa5df70ebdfc627bccb6b4c5b128aec6beb147740a91ca881f"}
Oct 30 20:58:49.625039 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.624963    5002 scope.go:122] "RemoveContainer" containerID="900d36b82d090acafa474456f7a3c91d019227a1a5a93853dc515666fb097353"
Oct 30 20:58:49.674943 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.674884    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:58:49.787795 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.787760    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-envoy\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.787966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.787817    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-podinfo\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.787966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.787860    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-data\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.787966 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.787942    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-token\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-token\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.788167 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788047    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-socket\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.788167 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788098    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-certs\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.788282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788169    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-kube-api-access-9fqf2\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-kube-api-access-9fqf2\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.788282 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788244    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-credential-socket\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.788385 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788281    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istiod-ca-cert\") pod \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\" (UID: \"e6ad1cb3-708e-48f5-abe2-be61ea83e237\") "
Oct 30 20:58:49.789110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788492    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-certs" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.789110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788701    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-data" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.789110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788940    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-socket" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.789110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.789031    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.789110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.789052    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.789110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.788975    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istiod-ca-cert" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:49.789110 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.789057    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-credential-socket" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.790518 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.790485    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-envoy" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.790811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.790761    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-podinfo" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:58:49.791287 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.791268    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-token" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:49.791456 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.791435    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-kube-api-access-9fqf2" pod "e6ad1cb3-708e-48f5-abe2-be61ea83e237" (UID: "e6ad1cb3-708e-48f5-abe2-be61ea83e237"). InnerVolumeSpecName "kube-api-access-9fqf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:49.889957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.889930    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.889957 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.889956    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.890111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.889965    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.890111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.889974    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqf2\" (UniqueName: \"kubernetes.io/projected/e6ad1cb3-708e-48f5-abe2-be61ea83e237-kube-api-access-9fqf2\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.890111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.889984    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.890111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.889991    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.890111 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:49.889999    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1cb3-708e-48f5-abe2-be61ea83e237-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:50.553052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.553027    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 20:58:50.632576 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.632547    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf" event={"ID":"e6ad1cb3-708e-48f5-abe2-be61ea83e237","Type":"ContainerDied","Data":"689311eb7584a5dc5cab2a1f7f080ddcae0b7baa20919de57f1fd4f0b9c36792"}
Oct 30 20:58:50.632708 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.632582    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"
Oct 30 20:58:50.632708 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.632598    5002 scope.go:122] "RemoveContainer" containerID="9cc42d1ebdac21aa5df70ebdfc627bccb6b4c5b128aec6beb147740a91ca881f"
Oct 30 20:58:50.787182 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.787153    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"]
Oct 30 20:58:50.834231 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.834170    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf"]
Oct 30 20:58:50.905361 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.905333    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"]
Oct 30 20:58:50.907149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.907118    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy"
Oct 30 20:58:50.907149 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.907147    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy"
Oct 30 20:58:50.907319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.907259    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy"
Oct 30 20:58:50.907319 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.907268    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy"
Oct 30 20:58:50.907621 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.907604    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy"
Oct 30 20:58:50.907685 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.907631    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" containerName="istio-proxy"
Oct 30 20:58:50.914593 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.914574    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:50.967662 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.967641    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-23491e-serving-cert\""
Oct 30 20:58:50.967882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:50.967860    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"]
Oct 30 20:58:51.003022 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.002989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.003144 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.003105    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf1e60-ab9b-414a-a360-820275adfbff-openshift-service-ca\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.003208 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.003162    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4cwg\" (UniqueName: \"kubernetes.io/projected/6dcf1e60-ab9b-414a-a360-820275adfbff-kube-api-access-d4cwg\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.003270 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.003211    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-metrics-certs\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.104713 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.104633    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.104713 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.104693    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf1e60-ab9b-414a-a360-820275adfbff-openshift-service-ca\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.104926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.104743    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4cwg\" (UniqueName: \"kubernetes.io/projected/6dcf1e60-ab9b-414a-a360-820275adfbff-kube-api-access-d4cwg\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.104926 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:51.104785    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-serving-cert: secret "maas-api-e2e-api-b-23491e-serving-cert" not found
Oct 30 20:58:51.104926 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.104799    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-metrics-certs\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.104926 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:58:51.104864    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls podName:6dcf1e60-ab9b-414a-a360-820275adfbff nodeName:}" failed. No retries permitted until 2026-10-30 20:58:51.60484614 +0000 UTC m=+2357.753897119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls") pod "maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt" (UID: "6dcf1e60-ab9b-414a-a360-820275adfbff") : secret "maas-api-e2e-api-b-23491e-serving-cert" not found
Oct 30 20:58:51.105351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.105331    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf1e60-ab9b-414a-a360-820275adfbff-openshift-service-ca\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.107344 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.107320    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-metrics-certs\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.137711 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.137683    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:51.163211 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.163165    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4cwg\" (UniqueName: \"kubernetes.io/projected/6dcf1e60-ab9b-414a-a360-820275adfbff-kube-api-access-d4cwg\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.315767 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.315732    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:51.488950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.488905    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:51.611796 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.611766    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.614145 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.614123    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls\") pod \"maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.829323 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.829265    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:51.967398 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:51.967370    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ad1cb3-708e-48f5-abe2-be61ea83e237" path="/var/lib/kubelet/pods/e6ad1cb3-708e-48f5-abe2-be61ea83e237/volumes"
Oct 30 20:58:52.210725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:52.210691    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"]
Oct 30 20:58:52.210916 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:58:52.210894    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcf1e60_ab9b_414a_a360_820275adfbff.slice/crio-c955d439019d5b967f6caf3f25ed9b5932afb975e5846723a9a8f4e30db84ad4 WatchSource:0}: Error finding container c955d439019d5b967f6caf3f25ed9b5932afb975e5846723a9a8f4e30db84ad4: Status 404 returned error can't find the container with id c955d439019d5b967f6caf3f25ed9b5932afb975e5846723a9a8f4e30db84ad4
Oct 30 20:58:52.212403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:52.212382    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:58:52.649243 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:52.649204    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt" event={"ID":"6dcf1e60-ab9b-414a-a360-820275adfbff","Type":"ContainerStarted","Data":"c955d439019d5b967f6caf3f25ed9b5932afb975e5846723a9a8f4e30db84ad4"}
Oct 30 20:58:53.657784 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:53.657748    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt" event={"ID":"6dcf1e60-ab9b-414a-a360-820275adfbff","Type":"ContainerStarted","Data":"a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c"}
Oct 30 20:58:53.658201 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:53.657801    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:58:53.776185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:53.776129    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt" podStartSLOduration=2.794731175 podStartE2EDuration="3.776116448s" totalImagesPullingTime="981.385273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:53.776086231 +0000 UTC m=+2359.925137220" watchObservedRunningTime="2026-10-30 20:58:53.776116448 +0000 UTC m=+2359.925167434"
Oct 30 20:58:56.067155 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:56.062378    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:58.748902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:58.748863    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:58.941352 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:58.941315    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:59.669367 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:58:59.669338    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 20:59:00.739524 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:00.739491    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:05.412098 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:05.412028    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:07.164055 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:07.164011    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:59:09.449219 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.449177    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"]
Oct 30 20:59:09.449774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.449380    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6" podUID="3e8f95c6-f0fa-44c5-a034-05450655a684" containerName="maas-api" containerID="cri-o://ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6" gracePeriod=30
Oct 30 20:59:09.601452 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.601403    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:09.777653 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.777605    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:59:09.782486 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.782456    5002 generic.go:356] "Generic (PLEG): container finished" podID="3e8f95c6-f0fa-44c5-a034-05450655a684" containerID="ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6" exitCode=0
Oct 30 20:59:09.782616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.782501    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"
Oct 30 20:59:09.782616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.782554    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6" event={"ID":"3e8f95c6-f0fa-44c5-a034-05450655a684","Type":"ContainerDied","Data":"ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6"}
Oct 30 20:59:09.782616 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.782609    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6" event={"ID":"3e8f95c6-f0fa-44c5-a034-05450655a684","Type":"ContainerDied","Data":"facb2bb1a2471282ceeb3dd58addf59a662150fefc712d82990341c29f812950"}
Oct 30 20:59:09.782736 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.782634    5002 scope.go:122] "RemoveContainer" containerID="ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6"
Oct 30 20:59:09.793687 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.793667    5002 scope.go:122] "RemoveContainer" containerID="ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6"
Oct 30 20:59:09.794028 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:09.794007    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6\": container with ID starting with ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6 not found: ID does not exist" containerID="ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6"
Oct 30 20:59:09.794107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.794039    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6"} err="failed to get container status \"ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6\": rpc error: code = NotFound desc = could not find container \"ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6\": container with ID starting with ac01d14afd2ac16ad377a0bb6311a550cff6ace1da30840a7fadd21c06843fb6 not found: ID does not exist"
Oct 30 20:59:09.843714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.838811    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-metrics-certs\") pod \"3e8f95c6-f0fa-44c5-a034-05450655a684\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") "
Oct 30 20:59:09.843714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.839035    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8f95c6-f0fa-44c5-a034-05450655a684-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8f95c6-f0fa-44c5-a034-05450655a684-openshift-service-ca\") pod \"3e8f95c6-f0fa-44c5-a034-05450655a684\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") "
Oct 30 20:59:09.843714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.839187    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8f95c6-f0fa-44c5-a034-05450655a684-kube-api-access-hnhwk\" (UniqueName: \"kubernetes.io/projected/3e8f95c6-f0fa-44c5-a034-05450655a684-kube-api-access-hnhwk\") pod \"3e8f95c6-f0fa-44c5-a034-05450655a684\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") "
Oct 30 20:59:09.843714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.839237    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls\") pod \"3e8f95c6-f0fa-44c5-a034-05450655a684\" (UID: \"3e8f95c6-f0fa-44c5-a034-05450655a684\") "
Oct 30 20:59:09.843714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.841278    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8f95c6-f0fa-44c5-a034-05450655a684-openshift-service-ca" pod "3e8f95c6-f0fa-44c5-a034-05450655a684" (UID: "3e8f95c6-f0fa-44c5-a034-05450655a684"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:09.843714 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.843686    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls" pod "3e8f95c6-f0fa-44c5-a034-05450655a684" (UID: "3e8f95c6-f0fa-44c5-a034-05450655a684"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:09.867556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.863524    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8f95c6-f0fa-44c5-a034-05450655a684-kube-api-access-hnhwk" pod "3e8f95c6-f0fa-44c5-a034-05450655a684" (UID: "3e8f95c6-f0fa-44c5-a034-05450655a684"). InnerVolumeSpecName "kube-api-access-hnhwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:09.867556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.863859    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-metrics-certs" pod "3e8f95c6-f0fa-44c5-a034-05450655a684" (UID: "3e8f95c6-f0fa-44c5-a034-05450655a684"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:09.942515 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.942480    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:09.948985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.945676    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8f95c6-f0fa-44c5-a034-05450655a684-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:09.948985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.945719    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnhwk\" (UniqueName: \"kubernetes.io/projected/3e8f95c6-f0fa-44c5-a034-05450655a684-kube-api-access-hnhwk\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:09.948985 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:09.945756    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8f95c6-f0fa-44c5-a034-05450655a684-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:10.200679 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:10.200645    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"]
Oct 30 20:59:10.205779 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:10.205757    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6"]
Oct 30 20:59:11.275881 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:11.275842    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:11.486299 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:11.486267    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:11.686568 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:11.686441    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:11.968909 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:11.968866    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8f95c6-f0fa-44c5-a034-05450655a684" path="/var/lib/kubelet/pods/3e8f95c6-f0fa-44c5-a034-05450655a684/volumes"
Oct 30 20:59:12.138120 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:12.138083    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:12.676215 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:12.676168    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f0723-85795bb986-88nb6" podUID="3e8f95c6-f0fa-44c5-a034-05450655a684" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.128:8443/health\": context deadline exceeded"
Oct 30 20:59:16.732638 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:16.732587    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:18.314277 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:18.314243    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:18.504710 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:18.504677    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:18.689882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:18.689808    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:21.882671 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:21.882637    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:26.082179 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.082143    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"]
Oct 30 20:59:26.082673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.082343    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8" podUID="3032376b-74fb-403f-8e88-ea7d10c7421f" containerName="maas-api" containerID="cri-o://3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa" gracePeriod=30
Oct 30 20:59:26.225426 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.225387    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:26.377041 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.377013    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:59:26.463862 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.463836    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3032376b-74fb-403f-8e88-ea7d10c7421f-kube-api-access-9dhwx\" (UniqueName: \"kubernetes.io/projected/3032376b-74fb-403f-8e88-ea7d10c7421f-kube-api-access-9dhwx\") pod \"3032376b-74fb-403f-8e88-ea7d10c7421f\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") "
Oct 30 20:59:26.463976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.463881    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3032376b-74fb-403f-8e88-ea7d10c7421f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3032376b-74fb-403f-8e88-ea7d10c7421f-openshift-service-ca\") pod \"3032376b-74fb-403f-8e88-ea7d10c7421f\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") "
Oct 30 20:59:26.463976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.463917    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls\") pod \"3032376b-74fb-403f-8e88-ea7d10c7421f\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") "
Oct 30 20:59:26.464152 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.464137    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-metrics-certs\") pod \"3032376b-74fb-403f-8e88-ea7d10c7421f\" (UID: \"3032376b-74fb-403f-8e88-ea7d10c7421f\") "
Oct 30 20:59:26.464288 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.464268    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3032376b-74fb-403f-8e88-ea7d10c7421f-openshift-service-ca" pod "3032376b-74fb-403f-8e88-ea7d10c7421f" (UID: "3032376b-74fb-403f-8e88-ea7d10c7421f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:26.464601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.464582    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3032376b-74fb-403f-8e88-ea7d10c7421f-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:26.466146 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.466123    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls" pod "3032376b-74fb-403f-8e88-ea7d10c7421f" (UID: "3032376b-74fb-403f-8e88-ea7d10c7421f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:26.466248 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.466224    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-metrics-certs" pod "3032376b-74fb-403f-8e88-ea7d10c7421f" (UID: "3032376b-74fb-403f-8e88-ea7d10c7421f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:26.466290 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.466232    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3032376b-74fb-403f-8e88-ea7d10c7421f-kube-api-access-9dhwx" pod "3032376b-74fb-403f-8e88-ea7d10c7421f" (UID: "3032376b-74fb-403f-8e88-ea7d10c7421f"). InnerVolumeSpecName "kube-api-access-9dhwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:26.565351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.565320    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:26.565351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.565347    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dhwx\" (UniqueName: \"kubernetes.io/projected/3032376b-74fb-403f-8e88-ea7d10c7421f-kube-api-access-9dhwx\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:26.565351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.565357    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3032376b-74fb-403f-8e88-ea7d10c7421f-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:26.911676 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.911647    5002 generic.go:356] "Generic (PLEG): container finished" podID="3032376b-74fb-403f-8e88-ea7d10c7421f" containerID="3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa" exitCode=0
Oct 30 20:59:26.911832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.911695    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"
Oct 30 20:59:26.911832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.911729    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8" event={"ID":"3032376b-74fb-403f-8e88-ea7d10c7421f","Type":"ContainerDied","Data":"3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa"}
Oct 30 20:59:26.911832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.911783    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8" event={"ID":"3032376b-74fb-403f-8e88-ea7d10c7421f","Type":"ContainerDied","Data":"9d760a279d75ff0afbe04d1e26528f37e2fd205f82d4decfaf8bdc18e68f73ea"}
Oct 30 20:59:26.911832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.911805    5002 scope.go:122] "RemoveContainer" containerID="3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa"
Oct 30 20:59:26.923475 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.923454    5002 scope.go:122] "RemoveContainer" containerID="3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa"
Oct 30 20:59:26.923790 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:26.923769    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa\": container with ID starting with 3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa not found: ID does not exist" containerID="3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa"
Oct 30 20:59:26.923868 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:26.923797    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa"} err="failed to get container status \"3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa\": rpc error: code = NotFound desc = could not find container \"3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa\": container with ID starting with 3ff8662e123b331739db40b34d7550fff3df706173c5c2140e4f2f3f54ac23fa not found: ID does not exist"
Oct 30 20:59:27.089261 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:27.089169    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"]
Oct 30 20:59:27.095574 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:27.095499    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a78bf4-7fcf4869b-gj5c8"]
Oct 30 20:59:27.856588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:27.856555    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:27.967697 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:27.967665    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3032376b-74fb-403f-8e88-ea7d10c7421f" path="/var/lib/kubelet/pods/3032376b-74fb-403f-8e88-ea7d10c7421f/volumes"
Oct 30 20:59:28.054688 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:28.054653    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:28.229948 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:28.229916    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:28.780768 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:28.780719    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:30.967430 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:30.967398    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"]
Oct 30 20:59:31.077594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:31.077559    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"]
Oct 30 20:59:31.378303 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:31.378234    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:31.552052 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:31.552011    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:33.194812 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:33.194775    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:34.350505 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:34.350472    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:59:34.363852 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:34.363829    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 20:59:34.577859 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:34.577817    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:35.967976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:35.967941    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw" podUID="cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" containerName="payload-processing" containerID="cri-o://a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95" gracePeriod=25
Oct 30 20:59:36.078209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.078173    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l" podUID="71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" containerName="payload-pre-processing" containerID="cri-o://0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14" gracePeriod=25
Oct 30 20:59:36.235837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.235764    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:59:36.340205 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.340181    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:59:36.373939 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.373910    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-kube-api-access-f99x5\" (UniqueName: \"kubernetes.io/projected/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-kube-api-access-f99x5\") pod \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") "
Oct 30 20:59:36.374204 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.374186    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-plugins-config-volume\") pod \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\" (UID: \"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba\") "
Oct 30 20:59:36.374601 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.374517    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-plugins-config-volume" pod "cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" (UID: "cf6cbd36-468d-4fa2-ad6d-279a319bc7ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:36.374794 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.374677    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.376105 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.376084    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-kube-api-access-f99x5" pod "cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" (UID: "cf6cbd36-468d-4fa2-ad6d-279a319bc7ba"). InnerVolumeSpecName "kube-api-access-f99x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:36.403431 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.403406    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:36.475494 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.475459    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-kube-api-access-l5pft\" (UniqueName: \"kubernetes.io/projected/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-kube-api-access-l5pft\") pod \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") "
Oct 30 20:59:36.475673 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.475608    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-plugins-config-volume\") pod \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\" (UID: \"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda\") "
Oct 30 20:59:36.475946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.475923    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99x5\" (UniqueName: \"kubernetes.io/projected/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba-kube-api-access-f99x5\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.476057 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.476039    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-plugins-config-volume" pod "71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" (UID: "71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:36.478363 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.478335    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-kube-api-access-l5pft" pod "71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" (UID: "71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda"). InnerVolumeSpecName "kube-api-access-l5pft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:36.576700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.576631    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5pft\" (UniqueName: \"kubernetes.io/projected/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-kube-api-access-l5pft\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.576700 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.576663    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.618719 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.618689    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"]
Oct 30 20:59:36.619248 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619235    5002 cpu_manager.go:414] "containerMap: removing container" podUID="71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" containerName="payload-pre-processing"
Oct 30 20:59:36.619295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619249    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" containerName="payload-pre-processing"
Oct 30 20:59:36.619295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619275    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3032376b-74fb-403f-8e88-ea7d10c7421f" containerName="maas-api"
Oct 30 20:59:36.619295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619280    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3032376b-74fb-403f-8e88-ea7d10c7421f" containerName="maas-api"
Oct 30 20:59:36.619295 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619290    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3e8f95c6-f0fa-44c5-a034-05450655a684" containerName="maas-api"
Oct 30 20:59:36.619416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619316    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8f95c6-f0fa-44c5-a034-05450655a684" containerName="maas-api"
Oct 30 20:59:36.619416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619339    5002 cpu_manager.go:414] "containerMap: removing container" podUID="cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" containerName="payload-processing"
Oct 30 20:59:36.619416 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619343    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" containerName="payload-processing"
Oct 30 20:59:36.619509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619434    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3032376b-74fb-403f-8e88-ea7d10c7421f" containerName="maas-api"
Oct 30 20:59:36.619509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619442    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8f95c6-f0fa-44c5-a034-05450655a684" containerName="maas-api"
Oct 30 20:59:36.619509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619450    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" containerName="payload-processing"
Oct 30 20:59:36.619509 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.619458    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" containerName="payload-pre-processing"
Oct 30 20:59:36.623119 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.623103    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.632896 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.632869    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"]
Oct 30 20:59:36.636681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.636666    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.696614 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.696579    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"]
Oct 30 20:59:36.739594 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.739330    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"]
Oct 30 20:59:36.786999 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.786970    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787020    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca7680e5-2ded-4caf-a212-2863026c6641-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787089    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787185 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787137    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787216    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca7680e5-2ded-4caf-a212-2863026c6641-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787256    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787287    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787307    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787323    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787383    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787401    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwkrr\" (UniqueName: \"kubernetes.io/projected/ca7680e5-2ded-4caf-a212-2863026c6641-kube-api-access-hwkrr\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787465    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787483    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787511    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7btm\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-kube-api-access-w7btm\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787557    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.787776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787592    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787648    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca7680e5-2ded-4caf-a212-2863026c6641-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.787776 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.787665    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.888964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.888878    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7btm\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-kube-api-access-w7btm\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.888964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.888920    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.888964 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.888941    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889233 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.888976    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca7680e5-2ded-4caf-a212-2863026c6641-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889233 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.888996    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889233 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889049    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889258    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca7680e5-2ded-4caf-a212-2863026c6641-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889306    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889305    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889403 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889375    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889434    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca7680e5-2ded-4caf-a212-2863026c6641-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889440    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889492    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889511    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889520    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889589    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889639 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889622    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889651    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889664    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889704    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwkrr\" (UniqueName: \"kubernetes.io/projected/ca7680e5-2ded-4caf-a212-2863026c6641-kube-api-access-hwkrr\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889796    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889829    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.889976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889901    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.889976 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.889907    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.890353 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.890081    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca7680e5-2ded-4caf-a212-2863026c6641-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.890418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.890371    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.890418 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.890396    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.890588 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.890564    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.891960 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.891933    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca7680e5-2ded-4caf-a212-2863026c6641-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.892716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.892695    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.892716 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.892710    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.892991 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.892971    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca7680e5-2ded-4caf-a212-2863026c6641-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.946307 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.946280    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca7680e5-2ded-4caf-a212-2863026c6641-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.946407 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.946392    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7btm\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-kube-api-access-w7btm\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.947401 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.947375    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fg56m\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:36.947496 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.947479    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwkrr\" (UniqueName: \"kubernetes.io/projected/ca7680e5-2ded-4caf-a212-2863026c6641-kube-api-access-hwkrr\") pod \"maas-default-gateway-openshift-default-6886b96f9-b6sv7\" (UID: \"ca7680e5-2ded-4caf-a212-2863026c6641\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:36.949280 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:36.949258    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:37.046866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.045463    5002 generic.go:356] "Generic (PLEG): container finished" podID="cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" containerID="a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95" exitCode=0
Oct 30 20:59:37.046866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.045771    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw" event={"ID":"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba","Type":"ContainerDied","Data":"a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95"}
Oct 30 20:59:37.046866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.045839    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw" event={"ID":"cf6cbd36-468d-4fa2-ad6d-279a319bc7ba","Type":"ContainerDied","Data":"777640ce44335f55a6a7b15a6ba9247c24cf6e55239ba9df6e0806d278425424"}
Oct 30 20:59:37.046866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.045864    5002 scope.go:122] "RemoveContainer" containerID="a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95"
Oct 30 20:59:37.046866 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.046022    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"
Oct 30 20:59:37.054830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.051559    5002 generic.go:356] "Generic (PLEG): container finished" podID="71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" containerID="0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14" exitCode=0
Oct 30 20:59:37.054830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.051659    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l" event={"ID":"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda","Type":"ContainerDied","Data":"0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14"}
Oct 30 20:59:37.054830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.051702    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l" event={"ID":"71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda","Type":"ContainerDied","Data":"7b0da0f484fe08e201e1b36f6fa0687100dfa281f470aae7e18fb3d996e0a373"}
Oct 30 20:59:37.054830 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.051797    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"
Oct 30 20:59:37.065774 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.065720    5002 scope.go:122] "RemoveContainer" containerID="a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95"
Oct 30 20:59:37.066799 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:37.066779    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95\": container with ID starting with a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95 not found: ID does not exist" containerID="a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95"
Oct 30 20:59:37.066892 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.066808    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95"} err="failed to get container status \"a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95\": rpc error: code = NotFound desc = could not find container \"a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95\": container with ID starting with a429c1dd52381f3d338cc36cd9ad8cc3ad14cd2ea7ad482dbc836ecda48e3d95 not found: ID does not exist"
Oct 30 20:59:37.066892 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.066824    5002 scope.go:122] "RemoveContainer" containerID="0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14"
Oct 30 20:59:37.078664 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.078645    5002 scope.go:122] "RemoveContainer" containerID="0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14"
Oct 30 20:59:37.078970 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:37.078947    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14\": container with ID starting with 0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14 not found: ID does not exist" containerID="0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14"
Oct 30 20:59:37.079049 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.078981    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14"} err="failed to get container status \"0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14\": rpc error: code = NotFound desc = could not find container \"0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14\": container with ID starting with 0b8a340f202b9bb2169394082c92c7c7f39c5e14ecffa4fea2cd98a9b258eb14 not found: ID does not exist"
Oct 30 20:59:37.116689 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:37.116639    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 20:59:37.214522 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.214487    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"]
Oct 30 20:59:37.222211 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.222186    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l"]
Oct 30 20:59:37.238125 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.238104    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:37.334209 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.334176    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"]
Oct 30 20:59:37.339473 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.339448    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw"]
Oct 30 20:59:37.391252 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:59:37.390177    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1286f6f_71b5_499b_80ae_aafcf82c9c8a.slice/crio-2bbd4401873dd21a18a090a355c2bc26249563f79f53fe0b33ffa46aa831c629 WatchSource:0}: Error finding container 2bbd4401873dd21a18a090a355c2bc26249563f79f53fe0b33ffa46aa831c629: Status 404 returned error can't find the container with id 2bbd4401873dd21a18a090a355c2bc26249563f79f53fe0b33ffa46aa831c629
Oct 30 20:59:37.391252 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.390254    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"]
Oct 30 20:59:37.394094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.393836    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.394094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.393925    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.394094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.393972    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.633084 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:59:37.633054    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7680e5_2ded_4caf_a212_2863026c6641.slice/crio-cc2be24b65a123f2121fdea52faf642a491e5f1710b4999fa752ae68e731b0f6 WatchSource:0}: Error finding container cc2be24b65a123f2121fdea52faf642a491e5f1710b4999fa752ae68e731b0f6: Status 404 returned error can't find the container with id cc2be24b65a123f2121fdea52faf642a491e5f1710b4999fa752ae68e731b0f6
Oct 30 20:59:37.634810 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.634787    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"]
Oct 30 20:59:37.636567 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.636506    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.636669 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.636609    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.636726 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.636668    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.774292 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.774255    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"]
Oct 30 20:59:37.779460 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.779439    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:37.843788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.842753    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"]
Oct 30 20:59:37.843788 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.843565    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-92f229-serving-cert\""
Oct 30 20:59:37.846013 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.845989    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:37.904822 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.904789    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-metrics-certs\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:37.905020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.904871    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:37.905086 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.905071    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81548445-8347-4e1b-9ec9-4d87ccc5df70-openshift-service-ca\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:37.905137 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.905110    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79cf\" (UniqueName: \"kubernetes.io/projected/81548445-8347-4e1b-9ec9-4d87ccc5df70-kube-api-access-c79cf\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:37.967100 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.967020    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda" path="/var/lib/kubelet/pods/71d69e2c-d227-4e0c-b4f0-8ee8e8c72bda/volumes"
Oct 30 20:59:37.967468 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:37.967454    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6cbd36-468d-4fa2-ad6d-279a319bc7ba" path="/var/lib/kubelet/pods/cf6cbd36-468d-4fa2-ad6d-279a319bc7ba/volumes"
Oct 30 20:59:38.012395 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.011952    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-metrics-certs\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.012395 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.012074    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.012395 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.012217    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81548445-8347-4e1b-9ec9-4d87ccc5df70-openshift-service-ca\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.012395 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.012261    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79cf\" (UniqueName: \"kubernetes.io/projected/81548445-8347-4e1b-9ec9-4d87ccc5df70-kube-api-access-c79cf\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.012941 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:38.012647    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-serving-cert: secret "maas-api-e2e-api-b-92f229-serving-cert" not found
Oct 30 20:59:38.012941 ip-10-0-141-29 kubenswrapper[5002]: E1030 20:59:38.012712    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls podName:81548445-8347-4e1b-9ec9-4d87ccc5df70 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:38.512691296 +0000 UTC m=+2404.661742275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls") pod "maas-api-e2e-api-b-92f229-5554477548-x2dd9" (UID: "81548445-8347-4e1b-9ec9-4d87ccc5df70") : secret "maas-api-e2e-api-b-92f229-serving-cert" not found
Oct 30 20:59:38.037554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.013412    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81548445-8347-4e1b-9ec9-4d87ccc5df70-openshift-service-ca\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.037554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.016052    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-metrics-certs\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.038090 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.038058    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:38.073159 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.073122    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79cf\" (UniqueName: \"kubernetes.io/projected/81548445-8347-4e1b-9ec9-4d87ccc5df70-kube-api-access-c79cf\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.073690 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.073665    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" event={"ID":"ca7680e5-2ded-4caf-a212-2863026c6641","Type":"ContainerStarted","Data":"af2750c44605b5bde7ab93d09be1cc8724cca1383973af4b85e26924c6756277"}
Oct 30 20:59:38.073854 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.073725    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" event={"ID":"ca7680e5-2ded-4caf-a212-2863026c6641","Type":"ContainerStarted","Data":"cc2be24b65a123f2121fdea52faf642a491e5f1710b4999fa752ae68e731b0f6"}
Oct 30 20:59:38.075840 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.075816    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" event={"ID":"a1286f6f-71b5-499b-80ae-aafcf82c9c8a","Type":"ContainerStarted","Data":"43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9"}
Oct 30 20:59:38.075962 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.075876    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" event={"ID":"a1286f6f-71b5-499b-80ae-aafcf82c9c8a","Type":"ContainerStarted","Data":"2bbd4401873dd21a18a090a355c2bc26249563f79f53fe0b33ffa46aa831c629"}
Oct 30 20:59:38.201020 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.200951    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podStartSLOduration=2.200932007 podStartE2EDuration="2.200932007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:38.196290392 +0000 UTC m=+2404.345341378" watchObservedRunningTime="2026-10-30 20:59:38.200932007 +0000 UTC m=+2404.349982995"
Oct 30 20:59:38.237681 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.236939    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:38.242554 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.241701    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:38.254617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.249732    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:38.254617 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.249803    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:38.330556 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.327068    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podStartSLOduration=2.327049475 podStartE2EDuration="2.327049475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:38.324001915 +0000 UTC m=+2404.473052913" watchObservedRunningTime="2026-10-30 20:59:38.327049475 +0000 UTC m=+2404.476100462"
Oct 30 20:59:38.430555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.430478    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:38.557882 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.557764    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.562221 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.562188    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls\") pod \"maas-api-e2e-api-b-92f229-5554477548-x2dd9\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.698977 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.698842    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:38.958986 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.958860    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:38.971590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.970803    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fg56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:59:38.971590 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:38.970883    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:59:39.171703 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:59:39.171660    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81548445_8347_4e1b_9ec9_4d87ccc5df70.slice/crio-2f49156b5c6bc6d964984ca78f8f43eb01ab92e98257df049918fc1a1313564c WatchSource:0}: Error finding container 2f49156b5c6bc6d964984ca78f8f43eb01ab92e98257df049918fc1a1313564c: Status 404 returned error can't find the container with id 2f49156b5c6bc6d964984ca78f8f43eb01ab92e98257df049918fc1a1313564c
Oct 30 20:59:39.174993 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:39.174971    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"]
Oct 30 20:59:39.247947 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:39.247911    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:39.248241 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:39.248211    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:39.965626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:39.964951    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fg56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:59:39.965626 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:39.965317    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:59:40.094946 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:40.094916    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9" event={"ID":"81548445-8347-4e1b-9ec9-4d87ccc5df70","Type":"ContainerStarted","Data":"2f49156b5c6bc6d964984ca78f8f43eb01ab92e98257df049918fc1a1313564c"}
Oct 30 20:59:40.239609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:40.238869    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:40.239609 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:40.238942    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:40.944559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:40.944374    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"]
Oct 30 20:59:40.966559 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:40.964639    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:40.966730 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:40.966661    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fg56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:59:40.966811 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:40.966721    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:59:41.002815 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.002788    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"]
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.031636    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-model-cache\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.031690    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-dshm\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.031783    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-home\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.031835    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tmp-dir\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.031902    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tls-certs\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.031932    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fq9x\" (UniqueName: \"kubernetes.io/projected/4cbdc811-65cc-4aba-8be5-5a0150bd4402-kube-api-access-5fq9x\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.053865    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bcef0e\"/\"kube-root-ca.crt\""
Oct 30 20:59:41.070336 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.053962    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bcef0e\"/\"openshift-service-ca.crt\""
Oct 30 20:59:41.082070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.080602    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:41.082070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.080823    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bcef0e\"/\"test-model-bcef0e-kserve-self-signed-certs\""
Oct 30 20:59:41.082070 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.080912    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bcef0e\"/\"default-dockercfg-vqdvz\""
Oct 30 20:59:41.177334 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.177287    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tmp-dir\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.177586 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.177388    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tls-certs\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.177586 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.177420    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fq9x\" (UniqueName: \"kubernetes.io/projected/4cbdc811-65cc-4aba-8be5-5a0150bd4402-kube-api-access-5fq9x\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.177725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.177606    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-model-cache\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.177725 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.177642    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-dshm\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.177837 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.177732    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-home\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.180569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.178176    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-home\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.180569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.178427    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tmp-dir\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.180569 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.178777    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-model-cache\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.181982 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.181799    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-dshm\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.182832 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.182122    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9" event={"ID":"81548445-8347-4e1b-9ec9-4d87ccc5df70","Type":"ContainerStarted","Data":"51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9"}
Oct 30 20:59:41.183525 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.183060    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:41.184606 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.184524    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tls-certs\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.242959 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.242913    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:41.243526 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.243483    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:41.267107 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.267042    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fq9x\" (UniqueName: \"kubernetes.io/projected/4cbdc811-65cc-4aba-8be5-5a0150bd4402-kube-api-access-5fq9x\") pod \"test-model-bcef0e-kserve-8db8cb6b9-8rq42\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") " pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.337351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.337278    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9" podStartSLOduration=3.523377885 podStartE2EDuration="4.337259437s" totalImagesPullingTime="813.881552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:41.335116707 +0000 UTC m=+2407.484167695" watchObservedRunningTime="2026-10-30 20:59:41.337259437 +0000 UTC m=+2407.486310424"
Oct 30 20:59:41.371498 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.371462    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:41.902421 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.902396    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"]
Oct 30 20:59:41.903884 ip-10-0-141-29 kubenswrapper[5002]: W1030 20:59:41.903859    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cbdc811_65cc_4aba_8be5_5a0150bd4402.slice/crio-d980d04f90a93117494f7b0d963ec5d10e6cfd3304efe130650b5347d0984a22 WatchSource:0}: Error finding container d980d04f90a93117494f7b0d963ec5d10e6cfd3304efe130650b5347d0984a22: Status 404 returned error can't find the container with id d980d04f90a93117494f7b0d963ec5d10e6cfd3304efe130650b5347d0984a22
Oct 30 20:59:41.949911 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.949873    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fg56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:59:41.950186 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:41.950154    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:59:42.195239 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.195169    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42" event={"ID":"4cbdc811-65cc-4aba-8be5-5a0150bd4402","Type":"ContainerStarted","Data":"056b63b37ef65911d932f270b6a29353a72fda7352e0f222c9c5d1a3c3350865"}
Oct 30 20:59:42.195415 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.195398    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42" event={"ID":"4cbdc811-65cc-4aba-8be5-5a0150bd4402","Type":"ContainerStarted","Data":"d980d04f90a93117494f7b0d963ec5d10e6cfd3304efe130650b5347d0984a22"}
Oct 30 20:59:42.265584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.264947    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:42.265584 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.265006    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:42.364195 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.364085    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42" podStartSLOduration=2.364062289 podStartE2EDuration="2.364062289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:42.357502756 +0000 UTC m=+2408.506553752" watchObservedRunningTime="2026-10-30 20:59:42.364062289 +0000 UTC m=+2408.513113277"
Oct 30 20:59:42.371729 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.371702    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:42.950555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.950132    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fg56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:59:42.950555 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:42.950193    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:59:43.017092 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:43.017055    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:43.247557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:43.245879    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:43.247557 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:43.245934    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:43.949876 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:43.949835    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fg56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:59:43.950360 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:43.949898    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:59:44.239038 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:44.238997    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:44.239238 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:44.239061    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:44.949545 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:44.949508    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fg56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:59:44.949728 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:44.949569    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:59:45.239392 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:45.239356    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:59:45.239950 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:45.239401    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:59:45.977508 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:45.977291    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:45.978737 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:45.978649    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:45.979761 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:45.979726    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 20:59:46.268351 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:46.268284    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:46.268827 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:46.268594    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:46.269355 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:46.269335    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7"
Oct 30 20:59:47.705409 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:47.705370    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:48.207154 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:48.207124    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 20:59:51.371800 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:51.371769    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:51.384257 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:51.384233    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:52.283902 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:52.283872    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 20:59:52.419022 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:52.418985    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:53.004094 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:53.004056    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 20:59:57.146903 ip-10-0-141-29 kubenswrapper[5002]: I1030 20:59:57.146857    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:00.212796 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.212758    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"]
Oct 30 21:00:00.217159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.217135    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.316416 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.316386    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"]
Oct 30 21:00:00.321950 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.321918    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:00:00.360959 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.360911    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ca693-092e-40db-8487-3ec2f13bc8e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.361116 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.361002    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4fc\" (UniqueName: \"kubernetes.io/projected/911ca693-092e-40db-8487-3ec2f13bc8e2-kube-api-access-rk4fc\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.461963 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.461919    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ca693-092e-40db-8487-3ec2f13bc8e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.462132 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.461993    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4fc\" (UniqueName: \"kubernetes.io/projected/911ca693-092e-40db-8487-3ec2f13bc8e2-kube-api-access-rk4fc\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.462526 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.462508    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ca693-092e-40db-8487-3ec2f13bc8e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.519705 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.519624    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4fc\" (UniqueName: \"kubernetes.io/projected/911ca693-092e-40db-8487-3ec2f13bc8e2-kube-api-access-rk4fc\") pod \"maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.532368 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.532333    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:00.943875 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:00:00.943831    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911ca693_092e_40db_8487_3ec2f13bc8e2.slice/crio-4c52d55a2f6f9d17df744d34123cc839acd73a7a72475e24acbb7802d4790395 WatchSource:0}: Error finding container 4c52d55a2f6f9d17df744d34123cc839acd73a7a72475e24acbb7802d4790395: Status 404 returned error can't find the container with id 4c52d55a2f6f9d17df744d34123cc839acd73a7a72475e24acbb7802d4790395
Oct 30 21:00:00.945131 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:00.945100    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"]
Oct 30 21:00:01.379228 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:01.379196    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" event={"ID":"911ca693-092e-40db-8487-3ec2f13bc8e2","Type":"ContainerStarted","Data":"4c52d55a2f6f9d17df744d34123cc839acd73a7a72475e24acbb7802d4790395"}
Oct 30 21:00:02.004477 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:02.004436    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:02.388813 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:02.388777    5002 generic.go:356] "Generic (PLEG): container finished" podID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerID="ba15859c2767379aa40b4ad231ce897d231fc3d9d5bcf82a880f7dd1d5d1c062" exitCode=60
Oct 30 21:00:02.389351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:02.388858    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" event={"ID":"911ca693-092e-40db-8487-3ec2f13bc8e2","Type":"ContainerDied","Data":"ba15859c2767379aa40b4ad231ce897d231fc3d9d5bcf82a880f7dd1d5d1c062"}
Oct 30 21:00:02.389351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:02.389113    5002 scope.go:122] "RemoveContainer" containerID="ba15859c2767379aa40b4ad231ce897d231fc3d9d5bcf82a880f7dd1d5d1c062"
Oct 30 21:00:03.358771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.356830    5002 patch_prober.go:29] interesting pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": read tcp 10.132.0.2:37322->10.132.0.149:15021: read: connection reset by peer" start-of-body=
Oct 30 21:00:03.358771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.356893    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": read tcp 10.132.0.2:37322->10.132.0.149:15021: read: connection reset by peer"
Oct 30 21:00:03.398961 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.398930    5002 generic.go:356] "Generic (PLEG): container finished" podID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerID="447071ee0325719f813a62c85a9af087568511b2d7c360edd6ad5f9b7a54b6a8" exitCode=60
Oct 30 21:00:03.399320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.399007    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" event={"ID":"911ca693-092e-40db-8487-3ec2f13bc8e2","Type":"ContainerDied","Data":"447071ee0325719f813a62c85a9af087568511b2d7c360edd6ad5f9b7a54b6a8"}
Oct 30 21:00:03.399320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.399069    5002 scope.go:122] "RemoveContainer" containerID="ba15859c2767379aa40b4ad231ce897d231fc3d9d5bcf82a880f7dd1d5d1c062"
Oct 30 21:00:03.399432 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.399399    5002 scope.go:122] "RemoveContainer" containerID="447071ee0325719f813a62c85a9af087568511b2d7c360edd6ad5f9b7a54b6a8"
Oct 30 21:00:03.399789 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:03.399745    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq_odh-ai-gateway-infra(911ca693-092e-40db-8487-3ec2f13bc8e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2"
Oct 30 21:00:03.401028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.401010    5002 generic.go:356] "Generic (PLEG): container finished" podID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerID="70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a" exitCode=0
Oct 30 21:00:03.401101 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.401041    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" event={"ID":"efe31d77-e37f-41b1-87fb-8a3eb5374f40","Type":"ContainerDied","Data":"70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a"}
Oct 30 21:00:03.401382 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.401370    5002 scope.go:122] "RemoveContainer" containerID="70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a"
Oct 30 21:00:03.402849 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.402818    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:00:03.402938 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.402897    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:00:03.402938 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.402930    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:00:03.736468 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:03.736402    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:04.432559 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:04.428901    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" event={"ID":"efe31d77-e37f-41b1-87fb-8a3eb5374f40","Type":"ContainerStarted","Data":"39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56"}
Oct 30 21:00:04.441589 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:04.435821    5002 scope.go:122] "RemoveContainer" containerID="447071ee0325719f813a62c85a9af087568511b2d7c360edd6ad5f9b7a54b6a8"
Oct 30 21:00:04.441589 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:04.436197    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq_odh-ai-gateway-infra(911ca693-092e-40db-8487-3ec2f13bc8e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2"
Oct 30 21:00:04.462937 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:04.462904    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 21:00:04.463159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:04.463134    5002 patch_prober.go:29] interesting pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:00:04.463214 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:04.463191    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 21:00:04.850259 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:04.850225    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:05.462488 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:05.462455    5002 patch_prober.go:29] interesting pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:00:05.462907 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:05.462499    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 21:00:06.180732 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.179204    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"]
Oct 30 21:00:06.187844 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.187818    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.246671 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.243434    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"]
Oct 30 21:00:06.267053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.267018    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.267239 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.267133    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.267357 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.267331    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9qr\" (UniqueName: \"kubernetes.io/projected/21132cde-72ba-403c-8490-85b1e5a7d43a-kube-api-access-ll9qr\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.284969 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.284942    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:06.368049 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.368023    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9qr\" (UniqueName: \"kubernetes.io/projected/21132cde-72ba-403c-8490-85b1e5a7d43a-kube-api-access-ll9qr\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.368228 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.368076    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.368228 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.368216    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.368806 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.368777    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.370715 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.370696    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.437109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.437037    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9qr\" (UniqueName: \"kubernetes.io/projected/21132cde-72ba-403c-8490-85b1e5a7d43a-kube-api-access-ll9qr\") pod \"maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.463028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.463002    5002 patch_prober.go:29] interesting pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:00:06.463374 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.463047    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 21:00:06.502034 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.502008    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:06.907625 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:06.907585    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"]
Oct 30 21:00:06.908782 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:00:06.908756    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21132cde_72ba_403c_8490_85b1e5a7d43a.slice/crio-9a7109e4a4018df274f58ad44dc42c43c243bb619e956c2429fd2688225f96e5 WatchSource:0}: Error finding container 9a7109e4a4018df274f58ad44dc42c43c243bb619e956c2429fd2688225f96e5: Status 404 returned error can't find the container with id 9a7109e4a4018df274f58ad44dc42c43c243bb619e956c2429fd2688225f96e5
Oct 30 21:00:07.044558 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:07.041174    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:07.144193 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:07.144152    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:00:07.462229 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:07.462203    5002 patch_prober.go:29] interesting pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:00:07.462484 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:07.462247    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 21:00:07.462484 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:07.462312    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx" event={"ID":"21132cde-72ba-403c-8490-85b1e5a7d43a","Type":"ContainerStarted","Data":"f8ff63f641593021d28aab3c34000ec2a8e2425f1fe0a87afd62f8168d0e2ffc"}
Oct 30 21:00:07.462484 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:07.462357    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx" event={"ID":"21132cde-72ba-403c-8490-85b1e5a7d43a","Type":"ContainerStarted","Data":"9a7109e4a4018df274f58ad44dc42c43c243bb619e956c2429fd2688225f96e5"}
Oct 30 21:00:07.598558 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:07.598496    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx" podStartSLOduration=1.598482569 podStartE2EDuration="1.598482569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:07.595668577 +0000 UTC m=+2433.744719565" watchObservedRunningTime="2026-10-30 21:00:07.598482569 +0000 UTC m=+2433.747533601"
Oct 30 21:00:08.462662 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:08.462630    5002 patch_prober.go:29] interesting pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:00:08.462815 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:08.462689    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 21:00:08.470775 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:08.470745    5002 generic.go:356] "Generic (PLEG): container finished" podID="21132cde-72ba-403c-8490-85b1e5a7d43a" containerID="f8ff63f641593021d28aab3c34000ec2a8e2425f1fe0a87afd62f8168d0e2ffc" exitCode=0
Oct 30 21:00:08.470932 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:08.470817    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx" event={"ID":"21132cde-72ba-403c-8490-85b1e5a7d43a","Type":"ContainerDied","Data":"f8ff63f641593021d28aab3c34000ec2a8e2425f1fe0a87afd62f8168d0e2ffc"}
Oct 30 21:00:09.509846 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.508956    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 21:00:09.509846 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.509791    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 21:00:09.511071 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.511051    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 21:00:09.625520 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.625497    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:09.707966 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.707941    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-xks-ca\") pod \"21132cde-72ba-403c-8490-85b1e5a7d43a\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") "
Oct 30 21:00:09.708102 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.707988    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-openshift-ca\") pod \"21132cde-72ba-403c-8490-85b1e5a7d43a\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") "
Oct 30 21:00:09.708102 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.708083    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21132cde-72ba-403c-8490-85b1e5a7d43a-kube-api-access-ll9qr\" (UniqueName: \"kubernetes.io/projected/21132cde-72ba-403c-8490-85b1e5a7d43a-kube-api-access-ll9qr\") pod \"21132cde-72ba-403c-8490-85b1e5a7d43a\" (UID: \"21132cde-72ba-403c-8490-85b1e5a7d43a\") "
Oct 30 21:00:09.708359 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.708331    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-openshift-ca" pod "21132cde-72ba-403c-8490-85b1e5a7d43a" (UID: "21132cde-72ba-403c-8490-85b1e5a7d43a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:09.708470 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.708459    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-openshift-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:09.710172 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.710144    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-xks-ca" pod "21132cde-72ba-403c-8490-85b1e5a7d43a" (UID: "21132cde-72ba-403c-8490-85b1e5a7d43a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:09.710260 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.710212    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21132cde-72ba-403c-8490-85b1e5a7d43a-kube-api-access-ll9qr" pod "21132cde-72ba-403c-8490-85b1e5a7d43a" (UID: "21132cde-72ba-403c-8490-85b1e5a7d43a"). InnerVolumeSpecName "kube-api-access-ll9qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:09.810086 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.810025    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21132cde-72ba-403c-8490-85b1e5a7d43a-maas-api-xks-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:09.810086 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:09.810057    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll9qr\" (UniqueName: \"kubernetes.io/projected/21132cde-72ba-403c-8490-85b1e5a7d43a-kube-api-access-ll9qr\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:10.485833 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:10.485797    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx" event={"ID":"21132cde-72ba-403c-8490-85b1e5a7d43a","Type":"ContainerDied","Data":"9a7109e4a4018df274f58ad44dc42c43c243bb619e956c2429fd2688225f96e5"}
Oct 30 21:00:10.486003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:10.485851    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a7109e4a4018df274f58ad44dc42c43c243bb619e956c2429fd2688225f96e5"
Oct 30 21:00:10.486003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:10.485880    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"
Oct 30 21:00:12.081880 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:12.081850    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:13.963107 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:13.963080    5002 scope.go:122] "RemoveContainer" containerID="447071ee0325719f813a62c85a9af087568511b2d7c360edd6ad5f9b7a54b6a8"
Oct 30 21:00:14.516832 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:14.516763    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" event={"ID":"911ca693-092e-40db-8487-3ec2f13bc8e2","Type":"ContainerStarted","Data":"263fe2b52cc3a046f4c1417cdf217e0b9492cd835e2fe8d061183aa975e3b6b0"}
Oct 30 21:00:14.634905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:14.634847    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" podStartSLOduration=14.634835177 podStartE2EDuration="14.634835177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:14.632340084 +0000 UTC m=+2440.781391070" watchObservedRunningTime="2026-10-30 21:00:14.634835177 +0000 UTC m=+2440.783886164"
Oct 30 21:00:15.527780 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:15.527738    5002 generic.go:356] "Generic (PLEG): container finished" podID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerID="263fe2b52cc3a046f4c1417cdf217e0b9492cd835e2fe8d061183aa975e3b6b0" exitCode=60
Oct 30 21:00:15.528222 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:15.527805    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" event={"ID":"911ca693-092e-40db-8487-3ec2f13bc8e2","Type":"ContainerDied","Data":"263fe2b52cc3a046f4c1417cdf217e0b9492cd835e2fe8d061183aa975e3b6b0"}
Oct 30 21:00:15.528222 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:15.527868    5002 scope.go:122] "RemoveContainer" containerID="447071ee0325719f813a62c85a9af087568511b2d7c360edd6ad5f9b7a54b6a8"
Oct 30 21:00:15.528334 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:15.528226    5002 scope.go:122] "RemoveContainer" containerID="263fe2b52cc3a046f4c1417cdf217e0b9492cd835e2fe8d061183aa975e3b6b0"
Oct 30 21:00:15.528638 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:15.528608    5002 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq_odh-ai-gateway-infra(911ca693-092e-40db-8487-3ec2f13bc8e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2"
Oct 30 21:00:15.645599 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:15.645573    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"]
Oct 30 21:00:16.703660 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.703637    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:16.786137 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.786103    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911ca693-092e-40db-8487-3ec2f13bc8e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ca693-092e-40db-8487-3ec2f13bc8e2-maas-api-service-ca\") pod \"911ca693-092e-40db-8487-3ec2f13bc8e2\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") "
Oct 30 21:00:16.786307 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.786267    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911ca693-092e-40db-8487-3ec2f13bc8e2-kube-api-access-rk4fc\" (UniqueName: \"kubernetes.io/projected/911ca693-092e-40db-8487-3ec2f13bc8e2-kube-api-access-rk4fc\") pod \"911ca693-092e-40db-8487-3ec2f13bc8e2\" (UID: \"911ca693-092e-40db-8487-3ec2f13bc8e2\") "
Oct 30 21:00:16.786433 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.786412    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911ca693-092e-40db-8487-3ec2f13bc8e2-maas-api-service-ca" pod "911ca693-092e-40db-8487-3ec2f13bc8e2" (UID: "911ca693-092e-40db-8487-3ec2f13bc8e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:16.786743 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.786724    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ca693-092e-40db-8487-3ec2f13bc8e2-maas-api-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:16.788503 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.788483    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911ca693-092e-40db-8487-3ec2f13bc8e2-kube-api-access-rk4fc" pod "911ca693-092e-40db-8487-3ec2f13bc8e2" (UID: "911ca693-092e-40db-8487-3ec2f13bc8e2"). InnerVolumeSpecName "kube-api-access-rk4fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:16.876597 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.876559    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:16.887478 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:16.887452    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4fc\" (UniqueName: \"kubernetes.io/projected/911ca693-092e-40db-8487-3ec2f13bc8e2-kube-api-access-rk4fc\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:17.213255 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:17.213215    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:17.547581 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:17.547487    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"
Oct 30 21:00:17.547581 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:17.547496    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq" event={"ID":"911ca693-092e-40db-8487-3ec2f13bc8e2","Type":"ContainerDied","Data":"4c52d55a2f6f9d17df744d34123cc839acd73a7a72475e24acbb7802d4790395"}
Oct 30 21:00:17.547801 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:17.547582    5002 scope.go:122] "RemoveContainer" containerID="263fe2b52cc3a046f4c1417cdf217e0b9492cd835e2fe8d061183aa975e3b6b0"
Oct 30 21:00:17.717926 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:17.717896    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"]
Oct 30 21:00:17.768124 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:17.768092    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a78ac-29889900-j6wlq"]
Oct 30 21:00:17.968012 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:17.967977    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" path="/var/lib/kubelet/pods/911ca693-092e-40db-8487-3ec2f13bc8e2/volumes"
Oct 30 21:00:19.162101 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:19.162063    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:21.860926 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:21.860878    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:22.031701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:22.031616    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:22.164123 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:22.164046    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"]
Oct 30 21:00:22.282370 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:22.282323    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:26.639357 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:26.639326    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:28.793024 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:28.792987    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"]
Oct 30 21:00:29.089410 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:29.089316    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:29.261676 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:29.261640    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:31.607724 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:31.607689    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:33.794252 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:33.794214    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" podUID="2290d2df-e80e-4348-93cb-8a09f6357953" containerName="payload-pre-processing" containerID="cri-o://a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244" gracePeriod=25
Oct 30 21:00:36.498061 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:36.498019    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:37.117575 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:37.117521    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:00:37.185513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.185458    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main" containerID="cri-o://5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0" gracePeriod=45
Oct 30 21:00:37.482453 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.482431    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.565519    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb61b044-bc78-49d1-9059-46e6919ac07f-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb61b044-bc78-49d1-9059-46e6919ac07f-tls-certs\") pod \"fb61b044-bc78-49d1-9059-46e6919ac07f\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") "
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.565642    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-model-cache\") pod \"fb61b044-bc78-49d1-9059-46e6919ac07f\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") "
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.565764    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-tmp-dir\") pod \"fb61b044-bc78-49d1-9059-46e6919ac07f\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") "
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.565802    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb61b044-bc78-49d1-9059-46e6919ac07f-kube-api-access-6v25m\" (UniqueName: \"kubernetes.io/projected/fb61b044-bc78-49d1-9059-46e6919ac07f-kube-api-access-6v25m\") pod \"fb61b044-bc78-49d1-9059-46e6919ac07f\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") "
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.565874    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-dshm\") pod \"fb61b044-bc78-49d1-9059-46e6919ac07f\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") "
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.565958    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-home\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-home\") pod \"fb61b044-bc78-49d1-9059-46e6919ac07f\" (UID: \"fb61b044-bc78-49d1-9059-46e6919ac07f\") "
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.566007    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-model-cache" pod "fb61b044-bc78-49d1-9059-46e6919ac07f" (UID: "fb61b044-bc78-49d1-9059-46e6919ac07f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.566588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.566144    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-tmp-dir" pod "fb61b044-bc78-49d1-9059-46e6919ac07f" (UID: "fb61b044-bc78-49d1-9059-46e6919ac07f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.567436 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.567109    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.567436 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.567152    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.567436 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.567406    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-home" pod "fb61b044-bc78-49d1-9059-46e6919ac07f" (UID: "fb61b044-bc78-49d1-9059-46e6919ac07f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.575235 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.575205    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-dshm" pod "fb61b044-bc78-49d1-9059-46e6919ac07f" (UID: "fb61b044-bc78-49d1-9059-46e6919ac07f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.580061 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.579983    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb61b044-bc78-49d1-9059-46e6919ac07f-kube-api-access-6v25m" pod "fb61b044-bc78-49d1-9059-46e6919ac07f" (UID: "fb61b044-bc78-49d1-9059-46e6919ac07f"). InnerVolumeSpecName "kube-api-access-6v25m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:37.580316 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.580286    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb61b044-bc78-49d1-9059-46e6919ac07f-tls-certs" pod "fb61b044-bc78-49d1-9059-46e6919ac07f" (UID: "fb61b044-bc78-49d1-9059-46e6919ac07f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:37.668513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.668480    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v25m\" (UniqueName: \"kubernetes.io/projected/fb61b044-bc78-49d1-9059-46e6919ac07f-kube-api-access-6v25m\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.668513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.668511    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.668513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.668520    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb61b044-bc78-49d1-9059-46e6919ac07f-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.668736 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.668544    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb61b044-bc78-49d1-9059-46e6919ac07f-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.683461 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.683422    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" podUID="2290d2df-e80e-4348-93cb-8a09f6357953" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:37.710385 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.710345    5002 generic.go:356] "Generic (PLEG): container finished" podID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerID="5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0" exitCode=0
Oct 30 21:00:37.710564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.710391    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"
Oct 30 21:00:37.710564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.710417    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" event={"ID":"fb61b044-bc78-49d1-9059-46e6919ac07f","Type":"ContainerDied","Data":"5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0"}
Oct 30 21:00:37.710564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.710471    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px" event={"ID":"fb61b044-bc78-49d1-9059-46e6919ac07f","Type":"ContainerDied","Data":"b0cf944903eb92c2d59c77dda5e40bfe56c16cfca0a4cdc0a767030d65871332"}
Oct 30 21:00:37.710564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.710489    5002 scope.go:122] "RemoveContainer" containerID="5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0"
Oct 30 21:00:37.724028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.724005    5002 scope.go:122] "RemoveContainer" containerID="5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0"
Oct 30 21:00:37.724330 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:37.724300    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0\": container with ID starting with 5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0 not found: ID does not exist" containerID="5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0"
Oct 30 21:00:37.724402 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.724339    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0"} err="failed to get container status \"5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0\": rpc error: code = NotFound desc = could not find container \"5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0\": container with ID starting with 5f9b257917653ca8642182926b696b410f672558e661edaab5ad1eadff2714e0 not found: ID does not exist"
Oct 30 21:00:37.880868 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.880829    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"]
Oct 30 21:00:37.883998 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.883978    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/simulator-w4-8e2fc1-kserve-78fc78bd5d-rv6px"]
Oct 30 21:00:37.968297 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:37.968227    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" path="/var/lib/kubelet/pods/fb61b044-bc78-49d1-9059-46e6919ac07f/volumes"
Oct 30 21:00:38.000575 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:38.000492    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:41.295364 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:41.295331    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:41.504248 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:41.504193    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:42.244917 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:42.244877    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:00:42.370983 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:42.370893    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:42.575323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:42.575234    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:46.807758 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:46.807711    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:47.683237 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:47.683194    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" podUID="2290d2df-e80e-4348-93cb-8a09f6357953" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:51.898208 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:51.898168    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:54.073690 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.073663    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 21:00:54.180115 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.180086    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"]
Oct 30 21:00:54.216055 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.216020    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2290d2df-e80e-4348-93cb-8a09f6357953-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2290d2df-e80e-4348-93cb-8a09f6357953-plugins-config-volume\") pod \"2290d2df-e80e-4348-93cb-8a09f6357953\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") "
Oct 30 21:00:54.216229 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.216102    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2290d2df-e80e-4348-93cb-8a09f6357953-kube-api-access-lr9mq\" (UniqueName: \"kubernetes.io/projected/2290d2df-e80e-4348-93cb-8a09f6357953-kube-api-access-lr9mq\") pod \"2290d2df-e80e-4348-93cb-8a09f6357953\" (UID: \"2290d2df-e80e-4348-93cb-8a09f6357953\") "
Oct 30 21:00:54.216489 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.216460    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2290d2df-e80e-4348-93cb-8a09f6357953-plugins-config-volume" pod "2290d2df-e80e-4348-93cb-8a09f6357953" (UID: "2290d2df-e80e-4348-93cb-8a09f6357953"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:54.216929 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.216814    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2290d2df-e80e-4348-93cb-8a09f6357953-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.218433 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.218406    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2290d2df-e80e-4348-93cb-8a09f6357953-kube-api-access-lr9mq" pod "2290d2df-e80e-4348-93cb-8a09f6357953" (UID: "2290d2df-e80e-4348-93cb-8a09f6357953"). InnerVolumeSpecName "kube-api-access-lr9mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:54.318098 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.318021    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr9mq\" (UniqueName: \"kubernetes.io/projected/2290d2df-e80e-4348-93cb-8a09f6357953-kube-api-access-lr9mq\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.440768 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.440731    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:54.615632 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.615503    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:54.869000 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.868920    5002 generic.go:356] "Generic (PLEG): container finished" podID="2290d2df-e80e-4348-93cb-8a09f6357953" containerID="a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244" exitCode=0
Oct 30 21:00:54.869000 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.868968    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"
Oct 30 21:00:54.869207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.869007    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" event={"ID":"2290d2df-e80e-4348-93cb-8a09f6357953","Type":"ContainerDied","Data":"a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244"}
Oct 30 21:00:54.869207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.869062    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j" event={"ID":"2290d2df-e80e-4348-93cb-8a09f6357953","Type":"ContainerDied","Data":"018bac28dae9a8f9d7159bc4f277a8591f07c13bb3a8492502b8e511ead3a3ee"}
Oct 30 21:00:54.869207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.869089    5002 scope.go:122] "RemoveContainer" containerID="a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244"
Oct 30 21:00:54.880923 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.880905    5002 scope.go:122] "RemoveContainer" containerID="a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244"
Oct 30 21:00:54.881172 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:54.881153    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244\": container with ID starting with a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244 not found: ID does not exist" containerID="a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244"
Oct 30 21:00:54.881245 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:54.881180    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244"} err="failed to get container status \"a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244\": rpc error: code = NotFound desc = could not find container \"a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244\": container with ID starting with a17c8f2ec31da1f1b46fcd7400a25b9a3afc2483bc569ecbf6f13bcaae4d7244 not found: ID does not exist"
Oct 30 21:00:55.037587 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:55.037556    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"]
Oct 30 21:00:55.041276 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:55.041246    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j"]
Oct 30 21:00:55.972055 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:55.972022    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2290d2df-e80e-4348-93cb-8a09f6357953" path="/var/lib/kubelet/pods/2290d2df-e80e-4348-93cb-8a09f6357953/volumes"
Oct 30 21:00:56.788568 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:56.788525    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:58.671932 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:58.671896    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:58.854630 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:58.854601    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:59.180515 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.180481    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg" podUID="8349d596-e27d-46f9-923a-e731adcf5292" containerName="payload-pre-processing" containerID="cri-o://dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85" gracePeriod=25
Oct 30 21:00:59.285339 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:00:59.285316    5002 logging.go:55] [core] [Channel #5101 SubChannel #5102]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.139:50052", ServerName: "10.132.0.139:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.139:50052: connect: connection refused"
Oct 30 21:00:59.459063 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.459040    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 21:00:59.585265 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.585233    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8349d596-e27d-46f9-923a-e731adcf5292-kube-api-access-5js49\" (UniqueName: \"kubernetes.io/projected/8349d596-e27d-46f9-923a-e731adcf5292-kube-api-access-5js49\") pod \"8349d596-e27d-46f9-923a-e731adcf5292\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") "
Oct 30 21:00:59.585411 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.585299    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8349d596-e27d-46f9-923a-e731adcf5292-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8349d596-e27d-46f9-923a-e731adcf5292-plugins-config-volume\") pod \"8349d596-e27d-46f9-923a-e731adcf5292\" (UID: \"8349d596-e27d-46f9-923a-e731adcf5292\") "
Oct 30 21:00:59.585795 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.585767    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8349d596-e27d-46f9-923a-e731adcf5292-plugins-config-volume" pod "8349d596-e27d-46f9-923a-e731adcf5292" (UID: "8349d596-e27d-46f9-923a-e731adcf5292"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:59.588111 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.588083    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8349d596-e27d-46f9-923a-e731adcf5292-kube-api-access-5js49" pod "8349d596-e27d-46f9-923a-e731adcf5292" (UID: "8349d596-e27d-46f9-923a-e731adcf5292"). InnerVolumeSpecName "kube-api-access-5js49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:59.686637 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.686606    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5js49\" (UniqueName: \"kubernetes.io/projected/8349d596-e27d-46f9-923a-e731adcf5292-kube-api-access-5js49\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.686637 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.686634    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8349d596-e27d-46f9-923a-e731adcf5292-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.740086 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.740061    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:59.907641 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.907615    5002 generic.go:356] "Generic (PLEG): container finished" podID="8349d596-e27d-46f9-923a-e731adcf5292" containerID="dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85" exitCode=0
Oct 30 21:00:59.907755 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.907655    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg" event={"ID":"8349d596-e27d-46f9-923a-e731adcf5292","Type":"ContainerDied","Data":"dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85"}
Oct 30 21:00:59.907755 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.907679    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"
Oct 30 21:00:59.907755 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.907692    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg" event={"ID":"8349d596-e27d-46f9-923a-e731adcf5292","Type":"ContainerDied","Data":"7403f83db3a54e76f1ebd19f34d4fc8ec3c49ed138293d6636640463a641fc91"}
Oct 30 21:00:59.907755 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.907718    5002 scope.go:122] "RemoveContainer" containerID="dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85"
Oct 30 21:00:59.917950 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.917932    5002 scope.go:122] "RemoveContainer" containerID="dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85"
Oct 30 21:00:59.918205 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:00:59.918182    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85\": container with ID starting with dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85 not found: ID does not exist" containerID="dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85"
Oct 30 21:00:59.918268 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:00:59.918212    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85"} err="failed to get container status \"dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85\": rpc error: code = NotFound desc = could not find container \"dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85\": container with ID starting with dd07788859edc836d1061bdbd36bcace2625e3a11c9a3bebf6b239f08b291b85 not found: ID does not exist"
Oct 30 21:01:00.073086 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:00.073022    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"]
Oct 30 21:01:00.076733 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:00.076713    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg"]
Oct 30 21:01:00.285939 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:00.285908    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg" podUID="8349d596-e27d-46f9-923a-e731adcf5292" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.139:50052\" within 1s: context deadline exceeded"
Oct 30 21:01:00.285939 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:01:00.285937    5002 logging.go:55] [core] [Channel #5101 SubChannel #5102]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.139:50052", ServerName: "10.132.0.139:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.139:50052: operation was canceled"
Oct 30 21:01:01.524424 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:01.522751    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:01.966451 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:01.966416    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8349d596-e27d-46f9-923a-e731adcf5292" path="/var/lib/kubelet/pods/8349d596-e27d-46f9-923a-e731adcf5292/volumes"
Oct 30 21:01:05.427347 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:05.427317    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:05.607847 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:05.607786    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:06.293390 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:06.293361    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:07.115272 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:07.115236    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:01:10.412631 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:10.412590    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:11.967199 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:11.967147    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:12.195827 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:12.195792    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:12.377986 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:12.377872    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:12.997403 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:12.997370    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:17.205300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:17.205157    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:19.609811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:19.609771    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:19.803926 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:19.803873    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:19.923395 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:19.923323    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"]
Oct 30 21:01:19.923664 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:19.923605    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy" containerID="cri-o://1c4a358103e222e5f47ad1f427b512893b32624110405a55fe9e3485cd7cb8b2" gracePeriod=30
Oct 30 21:01:20.014114 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:20.014069    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:20.742959 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:20.742926    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:21.327054 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.327014    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"]
Oct 30 21:01:21.327345 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.327313    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd" podUID="2c5325fc-fc92-499c-9845-ff8c6303dc31" containerName="maas-api" containerID="cri-o://b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d" gracePeriod=30
Oct 30 21:01:21.350059 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.350029    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:21.535396 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.535357    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:21.588152 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.588092    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 21:01:21.732006 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.730186    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:21.732388 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.732357    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls\") pod \"2c5325fc-fc92-499c-9845-ff8c6303dc31\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") "
Oct 30 21:01:21.732582 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.732556    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5325fc-fc92-499c-9845-ff8c6303dc31-kube-api-access-bcsrc\" (UniqueName: \"kubernetes.io/projected/2c5325fc-fc92-499c-9845-ff8c6303dc31-kube-api-access-bcsrc\") pod \"2c5325fc-fc92-499c-9845-ff8c6303dc31\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") "
Oct 30 21:01:21.732700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.732621    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5325fc-fc92-499c-9845-ff8c6303dc31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5325fc-fc92-499c-9845-ff8c6303dc31-openshift-service-ca\") pod \"2c5325fc-fc92-499c-9845-ff8c6303dc31\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") "
Oct 30 21:01:21.732700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.732664    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-metrics-certs\") pod \"2c5325fc-fc92-499c-9845-ff8c6303dc31\" (UID: \"2c5325fc-fc92-499c-9845-ff8c6303dc31\") "
Oct 30 21:01:21.747367 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.747331    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls" pod "2c5325fc-fc92-499c-9845-ff8c6303dc31" (UID: "2c5325fc-fc92-499c-9845-ff8c6303dc31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:21.747860 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.747757    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5325fc-fc92-499c-9845-ff8c6303dc31-openshift-service-ca" pod "2c5325fc-fc92-499c-9845-ff8c6303dc31" (UID: "2c5325fc-fc92-499c-9845-ff8c6303dc31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:21.762825 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.762770    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5325fc-fc92-499c-9845-ff8c6303dc31-kube-api-access-bcsrc" pod "2c5325fc-fc92-499c-9845-ff8c6303dc31" (UID: "2c5325fc-fc92-499c-9845-ff8c6303dc31"). InnerVolumeSpecName "kube-api-access-bcsrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:21.774790 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.774758    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-metrics-certs" pod "2c5325fc-fc92-499c-9845-ff8c6303dc31" (UID: "2c5325fc-fc92-499c-9845-ff8c6303dc31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:21.837391 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.837343    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:21.837391 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.837387    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcsrc\" (UniqueName: \"kubernetes.io/projected/2c5325fc-fc92-499c-9845-ff8c6303dc31-kube-api-access-bcsrc\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:21.837691 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.837410    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5325fc-fc92-499c-9845-ff8c6303dc31-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:21.837691 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:21.837431    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c5325fc-fc92-499c-9845-ff8c6303dc31-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:22.088497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.088468    5002 generic.go:356] "Generic (PLEG): container finished" podID="2c5325fc-fc92-499c-9845-ff8c6303dc31" containerID="b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d" exitCode=0
Oct 30 21:01:22.088689 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.088520    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"
Oct 30 21:01:22.088689 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.088565    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd" event={"ID":"2c5325fc-fc92-499c-9845-ff8c6303dc31","Type":"ContainerDied","Data":"b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d"}
Oct 30 21:01:22.088689 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.088631    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd" event={"ID":"2c5325fc-fc92-499c-9845-ff8c6303dc31","Type":"ContainerDied","Data":"c1780e4e9d2e3b72baa9f466fcfe389a5e4db4ec03c17fe7b0bdd749def8c5fa"}
Oct 30 21:01:22.088689 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.088659    5002 scope.go:122] "RemoveContainer" containerID="b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d"
Oct 30 21:01:22.102937 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.100481    5002 scope.go:122] "RemoveContainer" containerID="b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d"
Oct 30 21:01:22.103412 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:22.103374    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d\": container with ID starting with b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d not found: ID does not exist" containerID="b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d"
Oct 30 21:01:22.103516 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.103409    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d"} err="failed to get container status \"b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d\": rpc error: code = NotFound desc = could not find container \"b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d\": container with ID starting with b8cbab9acb1fddae797441b94b1db9cff9f8cd5e7a3e2fff09828561d856df3d not found: ID does not exist"
Oct 30 21:01:22.254921 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.254817    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"]
Oct 30 21:01:22.339271 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.339236    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5dc13e-b9565868d-kgtvd"]
Oct 30 21:01:22.489214 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.489176    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"]
Oct 30 21:01:22.489467 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.489442    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld" podUID="4a899c5a-3146-4b9a-b284-f2959ebe544c" containerName="maas-api" containerID="cri-o://8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5" gracePeriod=30
Oct 30 21:01:22.628999 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.628958    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:22.764008 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.763940    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 21:01:22.850082 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.850055    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a899c5a-3146-4b9a-b284-f2959ebe544c-kube-api-access-cp8qn\" (UniqueName: \"kubernetes.io/projected/4a899c5a-3146-4b9a-b284-f2959ebe544c-kube-api-access-cp8qn\") pod \"4a899c5a-3146-4b9a-b284-f2959ebe544c\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") "
Oct 30 21:01:22.850240 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.850193    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls\") pod \"4a899c5a-3146-4b9a-b284-f2959ebe544c\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") "
Oct 30 21:01:22.850240 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.850229    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a899c5a-3146-4b9a-b284-f2959ebe544c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a899c5a-3146-4b9a-b284-f2959ebe544c-openshift-service-ca\") pod \"4a899c5a-3146-4b9a-b284-f2959ebe544c\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") "
Oct 30 21:01:22.850368 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.850351    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-metrics-certs\") pod \"4a899c5a-3146-4b9a-b284-f2959ebe544c\" (UID: \"4a899c5a-3146-4b9a-b284-f2959ebe544c\") "
Oct 30 21:01:22.850650 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.850623    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a899c5a-3146-4b9a-b284-f2959ebe544c-openshift-service-ca" pod "4a899c5a-3146-4b9a-b284-f2959ebe544c" (UID: "4a899c5a-3146-4b9a-b284-f2959ebe544c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:22.851003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.850954    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a899c5a-3146-4b9a-b284-f2959ebe544c-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:22.852736 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.852712    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-metrics-certs" pod "4a899c5a-3146-4b9a-b284-f2959ebe544c" (UID: "4a899c5a-3146-4b9a-b284-f2959ebe544c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:22.852848 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.852818    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a899c5a-3146-4b9a-b284-f2959ebe544c-kube-api-access-cp8qn" pod "4a899c5a-3146-4b9a-b284-f2959ebe544c" (UID: "4a899c5a-3146-4b9a-b284-f2959ebe544c"). InnerVolumeSpecName "kube-api-access-cp8qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:22.852910 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.852848    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls" pod "4a899c5a-3146-4b9a-b284-f2959ebe544c" (UID: "4a899c5a-3146-4b9a-b284-f2959ebe544c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:22.952351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.952319    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:22.952351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.952348    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8qn\" (UniqueName: \"kubernetes.io/projected/4a899c5a-3146-4b9a-b284-f2959ebe544c-kube-api-access-cp8qn\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:22.952351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:22.952359    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a899c5a-3146-4b9a-b284-f2959ebe544c-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:23.005012 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.004950    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:23.097280 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.097196    5002 generic.go:356] "Generic (PLEG): container finished" podID="4a899c5a-3146-4b9a-b284-f2959ebe544c" containerID="8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5" exitCode=0
Oct 30 21:01:23.097280 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.097246    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"
Oct 30 21:01:23.097506 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.097277    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld" event={"ID":"4a899c5a-3146-4b9a-b284-f2959ebe544c","Type":"ContainerDied","Data":"8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5"}
Oct 30 21:01:23.097506 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.097327    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld" event={"ID":"4a899c5a-3146-4b9a-b284-f2959ebe544c","Type":"ContainerDied","Data":"170c2264c63bfd26dde8c475c3be3050f4b1336ce005a0ed47d8897c3a5fcf66"}
Oct 30 21:01:23.097506 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.097348    5002 scope.go:122] "RemoveContainer" containerID="8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5"
Oct 30 21:01:23.109913 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.109892    5002 scope.go:122] "RemoveContainer" containerID="8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5"
Oct 30 21:01:23.110177 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:23.110161    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5\": container with ID starting with 8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5 not found: ID does not exist" containerID="8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5"
Oct 30 21:01:23.110224 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.110185    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5"} err="failed to get container status \"8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5\": rpc error: code = NotFound desc = could not find container \"8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5\": container with ID starting with 8e40dd3ecbccd9b3925bd670108325b54a808e8a169328e66afad324255efff5 not found: ID does not exist"
Oct 30 21:01:23.265828 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.265796    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"]
Oct 30 21:01:23.295291 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.295262    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:23.318207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.318180    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e2fc1-c4d9598d6-5rbld"]
Oct 30 21:01:23.532772 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.532740    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"]
Oct 30 21:01:23.536051 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.536024    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5dc13e-wjtlx"]
Oct 30 21:01:23.952863 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.952786    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:23.968167 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.968140    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21132cde-72ba-403c-8490-85b1e5a7d43a" path="/var/lib/kubelet/pods/21132cde-72ba-403c-8490-85b1e5a7d43a/volumes"
Oct 30 21:01:23.968584 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.968568    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5325fc-fc92-499c-9845-ff8c6303dc31" path="/var/lib/kubelet/pods/2c5325fc-fc92-499c-9845-ff8c6303dc31/volumes"
Oct 30 21:01:23.968977 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:23.968963    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a899c5a-3146-4b9a-b284-f2959ebe544c" path="/var/lib/kubelet/pods/4a899c5a-3146-4b9a-b284-f2959ebe544c/volumes"
Oct 30 21:01:24.212507 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:24.212476    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:24.481129 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:24.481053    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:24.655444 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:24.655415    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:25.123894 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.123860    5002 generic.go:356] "Generic (PLEG): container finished" podID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerID="1c4a358103e222e5f47ad1f427b512893b32624110405a55fe9e3485cd7cb8b2" exitCode=0
Oct 30 21:01:25.124333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.123940    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" event={"ID":"88071f20-7d08-4bda-a6c6-65eccb3ab8e7","Type":"ContainerDied","Data":"1c4a358103e222e5f47ad1f427b512893b32624110405a55fe9e3485cd7cb8b2"}
Oct 30 21:01:25.124333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.124011    5002 scope.go:122] "RemoveContainer" containerID="d85cb981d53fc8363179c89b115c1acb69403a8c638a28a090bd0b0de71891f2"
Oct 30 21:01:25.151848 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.151812    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:25.212144 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.212123    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 21:01:25.281802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.281777    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-token\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-token\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.281917 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.281847    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-socket\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.281917 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.281877    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-envoy\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.281917 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.281906    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-data\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.282098 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282042    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istiod-ca-cert\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.282098 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282086    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-podinfo\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.282201 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282126    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-certs\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.282201 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282156    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-socket" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:25.282201 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282188    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-data" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:25.282201 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282197    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-kube-api-access-bfkk4\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-kube-api-access-bfkk4\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.282405 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282288    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-credential-socket\") pod \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\" (UID: \"88071f20-7d08-4bda-a6c6-65eccb3ab8e7\") "
Oct 30 21:01:25.282405 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282393    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-certs" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:25.282504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282433    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istiod-ca-cert" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:25.282682 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282653    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-credential-socket" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:25.282960 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282878    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.282960 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282899    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.282960 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282914    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.282960 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282927    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.282960 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.282940    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.284417 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.284394    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-podinfo" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:01:25.284583 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.284560    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-kube-api-access-bfkk4" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "kube-api-access-bfkk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:25.284695 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.284603    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-envoy" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:25.284695 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.284658    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-token" pod "88071f20-7d08-4bda-a6c6-65eccb3ab8e7" (UID: "88071f20-7d08-4bda-a6c6-65eccb3ab8e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:25.383955 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.383934    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.383955 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.383954    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkk4\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-kube-api-access-bfkk4\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.384076 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.383964    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:25.384076 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:25.383973    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88071f20-7d08-4bda-a6c6-65eccb3ab8e7-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:26.131517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.131483    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"
Oct 30 21:01:26.131949 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.131527    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k" event={"ID":"88071f20-7d08-4bda-a6c6-65eccb3ab8e7","Type":"ContainerDied","Data":"e747f228265b4bdefa4c512e6ddd8ba01de635b37d9032007c177f0a93225ee4"}
Oct 30 21:01:26.131949 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.131606    5002 scope.go:122] "RemoveContainer" containerID="1c4a358103e222e5f47ad1f427b512893b32624110405a55fe9e3485cd7cb8b2"
Oct 30 21:01:26.268932 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.268858    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:26.352849 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.352820    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"]
Oct 30 21:01:26.403192 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.403163    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k"]
Oct 30 21:01:26.515513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.515450    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:26.695013 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:26.694940    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:27.968384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:27.968346    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" path="/var/lib/kubelet/pods/88071f20-7d08-4bda-a6c6-65eccb3ab8e7/volumes"
Oct 30 21:01:29.504976 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:29.504944    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:29.695824 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:29.695783    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:29.905058 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:29.904984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:30.116194 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:30.115107    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:30.330493 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:30.327140    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:31.270885 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:31.270849    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"]
Oct 30 21:01:31.271642 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:31.271611    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy" containerID="cri-o://1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0" gracePeriod=30
Oct 30 21:01:31.672314 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:31.672215    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:31.855746 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:31.855711    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:32.142412 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:32.142379    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:33.371135 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:33.371104    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:01:34.714084 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:34.714046    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:36.533044 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.533018    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 21:01:36.633870 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.633792    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-credential-socket\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.633870 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.633848    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-kube-api-access-rjnhm\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-kube-api-access-rjnhm\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634089 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.633895    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istiod-ca-cert\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634089 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.633914    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-socket\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634089 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634025    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-data\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634089 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634048    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-credential-socket" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:36.634281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634103    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-podinfo\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634147    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-token\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-token\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634222    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-socket" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:36.634281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634223    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-envoy\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634468 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634291    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-certs\") pod \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\" (UID: \"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2\") "
Oct 30 21:01:36.634468 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634292    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istiod-ca-cert" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:36.634468 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634407    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-data" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:36.634739 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634579    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-certs" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:36.634850 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634832    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.634905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634855    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.634905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634868    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.634905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634877    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.634905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.634885    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.636863 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.636837    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-kube-api-access-rjnhm" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "kube-api-access-rjnhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:36.637171 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.637138    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-token" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:36.637293 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.637268    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-podinfo" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:01:36.637418 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.637344    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-envoy" pod "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" (UID: "a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:36.735997 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.735962    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.735997 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.735996    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.736176 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.736009    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.736176 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:36.736023    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnhm\" (UniqueName: \"kubernetes.io/projected/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2-kube-api-access-rjnhm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.121977 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:37.121924    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:01:37.209985 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.209948    5002 generic.go:356] "Generic (PLEG): container finished" podID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerID="1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0" exitCode=0
Oct 30 21:01:37.210157 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.210013    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"
Oct 30 21:01:37.210157 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.210056    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" event={"ID":"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2","Type":"ContainerDied","Data":"1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0"}
Oct 30 21:01:37.210157 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.210145    5002 scope.go:122] "RemoveContainer" containerID="1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0"
Oct 30 21:01:37.210589 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.210505    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc" event={"ID":"a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2","Type":"ContainerDied","Data":"8f3ff7b32e8ebff61fe05aa9d3fd49642932f0eda2a0e908ee46766a9470e084"}
Oct 30 21:01:37.224324 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.224302    5002 scope.go:122] "RemoveContainer" containerID="4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b"
Oct 30 21:01:37.233932 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.233912    5002 scope.go:122] "RemoveContainer" containerID="1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0"
Oct 30 21:01:37.234250 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:37.234222    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0\": container with ID starting with 1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0 not found: ID does not exist" containerID="1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0"
Oct 30 21:01:37.234323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.234261    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0"} err="failed to get container status \"1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0\": rpc error: code = NotFound desc = could not find container \"1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0\": container with ID starting with 1cc4556781cf5870868f322d54b359d799c0a20678f2658e2b9c320232d920f0 not found: ID does not exist"
Oct 30 21:01:37.234323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.234285    5002 scope.go:122] "RemoveContainer" containerID="4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b"
Oct 30 21:01:37.234665 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:37.234642    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b\": container with ID starting with 4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b not found: ID does not exist" containerID="4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b"
Oct 30 21:01:37.234746 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.234673    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b"} err="failed to get container status \"4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b\": rpc error: code = NotFound desc = could not find container \"4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b\": container with ID starting with 4e9aaed69504ea3277fd85740da7513417cc9b49acc44ea455a28d96493b236b not found: ID does not exist"
Oct 30 21:01:37.379987 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.379906    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"]
Oct 30 21:01:37.429630 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.429598    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc"]
Oct 30 21:01:37.967993 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:37.967962    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" path="/var/lib/kubelet/pods/a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2/volumes"
Oct 30 21:01:38.336235 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:38.336128    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:39.491184 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:39.491153    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:42.866812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:42.866785    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"]
Oct 30 21:01:42.975109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:42.975077    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"]
Oct 30 21:01:42.985819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:42.985787    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:43.268739 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:43.268684    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:43.446456 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:43.446424    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:44.235421 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:44.235389    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:47.867514 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:47.867462    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx" podUID="4298da21-8c66-4510-af57-9f4efc8fce69" containerName="payload-processing" containerID="cri-o://83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed" gracePeriod=25
Oct 30 21:01:47.975897 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:47.975852    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf" podUID="6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" containerName="payload-pre-processing" containerID="cri-o://747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb" gracePeriod=25
Oct 30 21:01:48.141471 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.141448    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 21:01:48.263784 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.263761    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4298da21-8c66-4510-af57-9f4efc8fce69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4298da21-8c66-4510-af57-9f4efc8fce69-plugins-config-volume\") pod \"4298da21-8c66-4510-af57-9f4efc8fce69\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") "
Oct 30 21:01:48.263999 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.263983    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4298da21-8c66-4510-af57-9f4efc8fce69-kube-api-access-d8gv9\" (UniqueName: \"kubernetes.io/projected/4298da21-8c66-4510-af57-9f4efc8fce69-kube-api-access-d8gv9\") pod \"4298da21-8c66-4510-af57-9f4efc8fce69\" (UID: \"4298da21-8c66-4510-af57-9f4efc8fce69\") "
Oct 30 21:01:48.264385 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.264145    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4298da21-8c66-4510-af57-9f4efc8fce69-plugins-config-volume" pod "4298da21-8c66-4510-af57-9f4efc8fce69" (UID: "4298da21-8c66-4510-af57-9f4efc8fce69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:48.264503 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.264482    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4298da21-8c66-4510-af57-9f4efc8fce69-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:48.266151 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.266126    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4298da21-8c66-4510-af57-9f4efc8fce69-kube-api-access-d8gv9" pod "4298da21-8c66-4510-af57-9f4efc8fce69" (UID: "4298da21-8c66-4510-af57-9f4efc8fce69"). InnerVolumeSpecName "kube-api-access-d8gv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:48.266778 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.266755    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 21:01:48.291434 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.291405    5002 generic.go:356] "Generic (PLEG): container finished" podID="4298da21-8c66-4510-af57-9f4efc8fce69" containerID="83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed" exitCode=0
Oct 30 21:01:48.291580 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.291456    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"
Oct 30 21:01:48.291580 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.291488    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx" event={"ID":"4298da21-8c66-4510-af57-9f4efc8fce69","Type":"ContainerDied","Data":"83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed"}
Oct 30 21:01:48.291580 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.291566    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx" event={"ID":"4298da21-8c66-4510-af57-9f4efc8fce69","Type":"ContainerDied","Data":"b6140347b38a93d16d0e660c332e3a643a967d083023f07976b9a4075a34470d"}
Oct 30 21:01:48.291760 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.291596    5002 scope.go:122] "RemoveContainer" containerID="83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed"
Oct 30 21:01:48.293384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.293360    5002 generic.go:356] "Generic (PLEG): container finished" podID="6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" containerID="747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb" exitCode=0
Oct 30 21:01:48.293492 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.293401    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"
Oct 30 21:01:48.293492 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.293404    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf" event={"ID":"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d","Type":"ContainerDied","Data":"747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb"}
Oct 30 21:01:48.293492 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.293441    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf" event={"ID":"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d","Type":"ContainerDied","Data":"132f2c0f500a4b3e0dafd3c377fead9315c5917c4c7b47fb406c7566d8eccc72"}
Oct 30 21:01:48.302781 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.302750    5002 scope.go:122] "RemoveContainer" containerID="83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed"
Oct 30 21:01:48.303043 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:48.303025    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed\": container with ID starting with 83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed not found: ID does not exist" containerID="83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed"
Oct 30 21:01:48.303112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.303051    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed"} err="failed to get container status \"83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed\": rpc error: code = NotFound desc = could not find container \"83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed\": container with ID starting with 83b521f53804023729bb924265aee00622a0f2f166faeac10352f394739b20ed not found: ID does not exist"
Oct 30 21:01:48.303112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.303067    5002 scope.go:122] "RemoveContainer" containerID="747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb"
Oct 30 21:01:48.313716 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.313696    5002 scope.go:122] "RemoveContainer" containerID="747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb"
Oct 30 21:01:48.313984 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:48.313967    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb\": container with ID starting with 747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb not found: ID does not exist" containerID="747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb"
Oct 30 21:01:48.314034 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.313994    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb"} err="failed to get container status \"747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb\": rpc error: code = NotFound desc = could not find container \"747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb\": container with ID starting with 747461a1c9bf82036d7937af64c093d4e664a048bae1d969cb08a719751c5ccb not found: ID does not exist"
Oct 30 21:01:48.365088 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.365067    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-kube-api-access-dksdj\" (UniqueName: \"kubernetes.io/projected/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-kube-api-access-dksdj\") pod \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") "
Oct 30 21:01:48.365202 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.365117    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-plugins-config-volume\") pod \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\" (UID: \"6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d\") "
Oct 30 21:01:48.365514 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.365481    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-plugins-config-volume" pod "6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" (UID: "6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:48.365638 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.365581    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8gv9\" (UniqueName: \"kubernetes.io/projected/4298da21-8c66-4510-af57-9f4efc8fce69-kube-api-access-d8gv9\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:48.367153 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.367131    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-kube-api-access-dksdj" pod "6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" (UID: "6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d"). InnerVolumeSpecName "kube-api-access-dksdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:48.467244 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.467213    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dksdj\" (UniqueName: \"kubernetes.io/projected/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-kube-api-access-dksdj\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:48.467244 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.467240    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:48.547752 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.547721    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"]
Oct 30 21:01:48.553038 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.553013    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx"]
Oct 30 21:01:48.716635 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.716614    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"]
Oct 30 21:01:48.719626 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.719578    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf"]
Oct 30 21:01:48.835667 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:48.835620    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:49.967305 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:49.967272    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4298da21-8c66-4510-af57-9f4efc8fce69" path="/var/lib/kubelet/pods/4298da21-8c66-4510-af57-9f4efc8fce69/volumes"
Oct 30 21:01:49.967703 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:49.967687    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" path="/var/lib/kubelet/pods/6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d/volumes"
Oct 30 21:01:53.164102 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:53.164048    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:53.353017 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:53.352988    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:53.525431 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:53.525394    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:55.807107 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:55.807071    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:01:57.444905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.444867    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"]
Oct 30 21:01:57.445306 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.445060    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7" podUID="75b1e9f0-85ae-4b8c-82d0-85eda9315a07" containerName="maas-api" containerID="cri-o://773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c" gracePeriod=30
Oct 30 21:01:57.581213 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.581178    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:57.721362 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.721338    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 21:01:57.880186 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.880154    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls\") pod \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") "
Oct 30 21:01:57.880386 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.880199    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-metrics-certs\") pod \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") "
Oct 30 21:01:57.880386 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.880243    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-kube-api-access-nhtck\" (UniqueName: \"kubernetes.io/projected/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-kube-api-access-nhtck\") pod \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") "
Oct 30 21:01:57.880500 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.880432    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-openshift-service-ca\") pod \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\" (UID: \"75b1e9f0-85ae-4b8c-82d0-85eda9315a07\") "
Oct 30 21:01:57.881130 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.881105    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-openshift-service-ca" pod "75b1e9f0-85ae-4b8c-82d0-85eda9315a07" (UID: "75b1e9f0-85ae-4b8c-82d0-85eda9315a07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:57.882963 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.882938    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-kube-api-access-nhtck" pod "75b1e9f0-85ae-4b8c-82d0-85eda9315a07" (UID: "75b1e9f0-85ae-4b8c-82d0-85eda9315a07"). InnerVolumeSpecName "kube-api-access-nhtck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:57.883773 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.883753    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls" pod "75b1e9f0-85ae-4b8c-82d0-85eda9315a07" (UID: "75b1e9f0-85ae-4b8c-82d0-85eda9315a07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:57.885339 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.885319    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-metrics-certs" pod "75b1e9f0-85ae-4b8c-82d0-85eda9315a07" (UID: "75b1e9f0-85ae-4b8c-82d0-85eda9315a07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:57.981926 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.981893    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:57.982164 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.982149    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:57.982255 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.982243    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtck\" (UniqueName: \"kubernetes.io/projected/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-kube-api-access-nhtck\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:57.982335 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:57.982325    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b1e9f0-85ae-4b8c-82d0-85eda9315a07-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:58.407456 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.407349    5002 generic.go:356] "Generic (PLEG): container finished" podID="75b1e9f0-85ae-4b8c-82d0-85eda9315a07" containerID="773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c" exitCode=0
Oct 30 21:01:58.407456 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.407406    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"
Oct 30 21:01:58.407756 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.407402    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7" event={"ID":"75b1e9f0-85ae-4b8c-82d0-85eda9315a07","Type":"ContainerDied","Data":"773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c"}
Oct 30 21:01:58.407756 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.407579    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7" event={"ID":"75b1e9f0-85ae-4b8c-82d0-85eda9315a07","Type":"ContainerDied","Data":"1bdef8ae7aad713415ed54bd5cc8c1731a702c5a2b11746d6160f7482fda8afb"}
Oct 30 21:01:58.407756 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.407625    5002 scope.go:122] "RemoveContainer" containerID="773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c"
Oct 30 21:01:58.421069 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.421047    5002 scope.go:122] "RemoveContainer" containerID="773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c"
Oct 30 21:01:58.421418 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:01:58.421389    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c\": container with ID starting with 773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c not found: ID does not exist" containerID="773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c"
Oct 30 21:01:58.421491 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.421453    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c"} err="failed to get container status \"773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c\": rpc error: code = NotFound desc = could not find container \"773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c\": container with ID starting with 773675fef03f129118a6809f00e61bcb3ce0cc24c5ff93aa6fe303fa5e72506c not found: ID does not exist"
Oct 30 21:01:58.574358 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.574319    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"]
Oct 30 21:01:58.579238 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:58.579211    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2d96-5c748cd5dc-qc2j7"]
Oct 30 21:01:59.082852 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:59.082789    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:59.344749 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:59.344641    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:59.522984 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:59.522933    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:59.967055 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:01:59.967020    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b1e9f0-85ae-4b8c-82d0-85eda9315a07" path="/var/lib/kubelet/pods/75b1e9f0-85ae-4b8c-82d0-85eda9315a07/volumes"
Oct 30 21:02:00.074281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:00.074249    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:04.281156 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:04.281124    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:05.753965 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:05.753926    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:05.939568 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:05.939495    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:06.067074 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:06.066993    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"]
Oct 30 21:02:06.067278 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:06.067249    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy" containerID="cri-o://640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406" gracePeriod=30
Oct 30 21:02:06.121541 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:06.121509    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:07.116220 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:02:07.116184    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:02:08.851132 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851094    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"]
Oct 30 21:02:08.851727 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851712    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4a899c5a-3146-4b9a-b284-f2959ebe544c" containerName="maas-api"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851728    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a899c5a-3146-4b9a-b284-f2959ebe544c" containerName="maas-api"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851742    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851749    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851759    5002 cpu_manager.go:414] "containerMap: removing container" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851768    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851778    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" containerName="payload-pre-processing"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851783    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" containerName="payload-pre-processing"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851791    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2290d2df-e80e-4348-93cb-8a09f6357953" containerName="payload-pre-processing"
Oct 30 21:02:08.851802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851796    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2290d2df-e80e-4348-93cb-8a09f6357953" containerName="payload-pre-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851809    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851815    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851822    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4298da21-8c66-4510-af57-9f4efc8fce69" containerName="payload-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851827    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4298da21-8c66-4510-af57-9f4efc8fce69" containerName="payload-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851840    5002 cpu_manager.go:414] "containerMap: removing container" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851848    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851855    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2c5325fc-fc92-499c-9845-ff8c6303dc31" containerName="maas-api"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851860    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5325fc-fc92-499c-9845-ff8c6303dc31" containerName="maas-api"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851871    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851875    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851882    5002 cpu_manager.go:414] "containerMap: removing container" podUID="75b1e9f0-85ae-4b8c-82d0-85eda9315a07" containerName="maas-api"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851886    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b1e9f0-85ae-4b8c-82d0-85eda9315a07" containerName="maas-api"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851894    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8349d596-e27d-46f9-923a-e731adcf5292" containerName="payload-pre-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851898    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8349d596-e27d-46f9-923a-e731adcf5292" containerName="payload-pre-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851906    5002 cpu_manager.go:414] "containerMap: removing container" podUID="21132cde-72ba-403c-8490-85b1e5a7d43a" containerName="revoke-keys"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851911    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21132cde-72ba-403c-8490-85b1e5a7d43a" containerName="revoke-keys"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851935    5002 cpu_manager.go:414] "containerMap: removing container" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.851939    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852036    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852046    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a899c5a-3146-4b9a-b284-f2959ebe544c" containerName="maas-api"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852052    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb61b044-bc78-49d1-9059-46e6919ac07f" containerName="main"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852059    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852065    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5c2f4b-d17a-4d11-8ad8-f5b91a17de3d" containerName="payload-pre-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852072    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2290d2df-e80e-4348-93cb-8a09f6357953" containerName="payload-pre-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852085    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852095    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4298da21-8c66-4510-af57-9f4efc8fce69" containerName="payload-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852101    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5325fc-fc92-499c-9845-ff8c6303dc31" containerName="maas-api"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852109    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92c5d8c-4387-4c48-ad67-a6a6d03d2ef2" containerName="istio-proxy"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852115    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b1e9f0-85ae-4b8c-82d0-85eda9315a07" containerName="maas-api"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852120    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8349d596-e27d-46f9-923a-e731adcf5292" containerName="payload-pre-processing"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852126    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="21132cde-72ba-403c-8490-85b1e5a7d43a" containerName="revoke-keys"
Oct 30 21:02:08.852243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.852131    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:02:08.857104 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.857086    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:08.910472 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.910441    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2fa881-serving-cert\""
Oct 30 21:02:08.926429 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.926400    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prw79\" (UniqueName: \"kubernetes.io/projected/114a89a2-44c3-4622-90a3-827fd3651162-kube-api-access-prw79\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:08.926637 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.926615    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-metrics-certs\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:08.926718 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.926655    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:08.926768 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.926714    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114a89a2-44c3-4622-90a3-827fd3651162-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:08.964605 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:08.964577    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"]
Oct 30 21:02:09.029421 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.029384    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prw79\" (UniqueName: \"kubernetes.io/projected/114a89a2-44c3-4622-90a3-827fd3651162-kube-api-access-prw79\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.029758 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.029582    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-metrics-certs\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.029758 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.029616    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.029758 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.029640    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114a89a2-44c3-4622-90a3-827fd3651162-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.030423 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.030396    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114a89a2-44c3-4622-90a3-827fd3651162-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.030674 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:02:09.030657    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-serving-cert: secret "maas-api-e2e-inf-a-2fa881-serving-cert" not found
Oct 30 21:02:09.030826 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:02:09.030815    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls podName:114a89a2-44c3-4622-90a3-827fd3651162 nodeName:}" failed. No retries permitted until 2026-10-30 21:02:09.530796485 +0000 UTC m=+2555.679847452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls") pod "maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8" (UID: "114a89a2-44c3-4622-90a3-827fd3651162") : secret "maas-api-e2e-inf-a-2fa881-serving-cert" not found
Oct 30 21:02:09.033171 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.033104    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-metrics-certs\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.087615 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.087586    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prw79\" (UniqueName: \"kubernetes.io/projected/114a89a2-44c3-4622-90a3-827fd3651162-kube-api-access-prw79\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.132863 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.132786    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:09.310260 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.310230    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:09.490940 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.490905    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:09.536287 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.536252    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.539162 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.539139    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls\") pod \"maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:09.771412 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:09.771313    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:10.155426 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:10.155394    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"]
Oct 30 21:02:10.155928 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:02:10.155906    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114a89a2_44c3_4622_90a3_827fd3651162.slice/crio-5ab889ca8c7c469b72062d91c6e6f8bb2db96268acc992338f14d434dc2b7460 WatchSource:0}: Error finding container 5ab889ca8c7c469b72062d91c6e6f8bb2db96268acc992338f14d434dc2b7460: Status 404 returned error can't find the container with id 5ab889ca8c7c469b72062d91c6e6f8bb2db96268acc992338f14d434dc2b7460
Oct 30 21:02:10.488889 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:10.488849    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8" event={"ID":"114a89a2-44c3-4622-90a3-827fd3651162","Type":"ContainerStarted","Data":"5ab889ca8c7c469b72062d91c6e6f8bb2db96268acc992338f14d434dc2b7460"}
Oct 30 21:02:11.351187 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.351158    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 21:02:11.461400 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461366    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-token\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-token\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461600 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461447    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4d728a-78c0-405c-9907-6df33d6bd997-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4d728a-78c0-405c-9907-6df33d6bd997-istiod-ca-cert\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461600 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461526    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-envoy\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461684 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461614    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-credential-socket\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461733 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461685    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-kube-api-access-9ctmt\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-kube-api-access-9ctmt\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461733 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461721    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-data\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461820 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461779    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-socket\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461877 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461821    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-podinfo\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461877 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461872    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-certs\") pod \"2a4d728a-78c0-405c-9907-6df33d6bd997\" (UID: \"2a4d728a-78c0-405c-9907-6df33d6bd997\") "
Oct 30 21:02:11.461988 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461881    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-credential-socket" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:11.461988 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.461902    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4d728a-78c0-405c-9907-6df33d6bd997-istiod-ca-cert" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:11.462097 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.462083    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-socket" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:11.462298 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.462257    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-data" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:11.462470 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.462444    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-certs" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:11.462816 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.462462    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.462923 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.462821    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.462923 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.462839    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4d728a-78c0-405c-9907-6df33d6bd997-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.462923 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.462854    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.464282 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.464262    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-envoy" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:11.464282 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.464266    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-token" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:11.464471 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.464279    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-podinfo" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:02:11.464471 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.464401    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-kube-api-access-9ctmt" pod "2a4d728a-78c0-405c-9907-6df33d6bd997" (UID: "2a4d728a-78c0-405c-9907-6df33d6bd997"). InnerVolumeSpecName "kube-api-access-9ctmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:11.497632 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.497593    5002 generic.go:356] "Generic (PLEG): container finished" podID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerID="640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406" exitCode=0
Oct 30 21:02:11.497812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.497647    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"
Oct 30 21:02:11.497812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.497683    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" event={"ID":"2a4d728a-78c0-405c-9907-6df33d6bd997","Type":"ContainerDied","Data":"640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406"}
Oct 30 21:02:11.497812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.497743    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45" event={"ID":"2a4d728a-78c0-405c-9907-6df33d6bd997","Type":"ContainerDied","Data":"3324b921a027fef7c332780ba75b6cc9956db6acbed22508122b4b72deb0aa30"}
Oct 30 21:02:11.497812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.497774    5002 scope.go:122] "RemoveContainer" containerID="640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406"
Oct 30 21:02:11.500163 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.499623    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8" event={"ID":"114a89a2-44c3-4622-90a3-827fd3651162","Type":"ContainerStarted","Data":"d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125"}
Oct 30 21:02:11.500163 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.499682    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:11.511041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.511021    5002 scope.go:122] "RemoveContainer" containerID="cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4"
Oct 30 21:02:11.522143 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.522110    5002 scope.go:122] "RemoveContainer" containerID="640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406"
Oct 30 21:02:11.522495 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:02:11.522465    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406\": container with ID starting with 640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406 not found: ID does not exist" containerID="640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406"
Oct 30 21:02:11.522612 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.522511    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406"} err="failed to get container status \"640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406\": rpc error: code = NotFound desc = could not find container \"640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406\": container with ID starting with 640c8edf48c42cf04518be6d0a4ea9603fc100ab36cdb365f581db50ddb5d406 not found: ID does not exist"
Oct 30 21:02:11.522612 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.522553    5002 scope.go:122] "RemoveContainer" containerID="cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4"
Oct 30 21:02:11.523701 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:02:11.523639    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4\": container with ID starting with cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4 not found: ID does not exist" containerID="cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4"
Oct 30 21:02:11.523701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.523672    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4"} err="failed to get container status \"cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4\": rpc error: code = NotFound desc = could not find container \"cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4\": container with ID starting with cd6d911b743a6c2f1471859d65ed9e7ce914ddf1dce4d734327db5ba105bf8b4 not found: ID does not exist"
Oct 30 21:02:11.563840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.563802    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.563840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.563838    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.563840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.563850    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.564103 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.563859    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctmt\" (UniqueName: \"kubernetes.io/projected/2a4d728a-78c0-405c-9907-6df33d6bd997-kube-api-access-9ctmt\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.564103 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.563869    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4d728a-78c0-405c-9907-6df33d6bd997-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.640376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.640256    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8" podStartSLOduration=2.7592352829999998 podStartE2EDuration="3.640242225s" totalImagesPullingTime="881.006942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:11.637704852 +0000 UTC m=+2557.786755841" watchObservedRunningTime="2026-10-30 21:02:11.640242225 +0000 UTC m=+2557.789293212"
Oct 30 21:02:11.753478 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.753432    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"]
Oct 30 21:02:11.756845 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.756810    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45"]
Oct 30 21:02:11.967573 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:11.967519    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" path="/var/lib/kubelet/pods/2a4d728a-78c0-405c-9907-6df33d6bd997/volumes"
Oct 30 21:02:14.194807 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:14.194771    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:17.513904 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:17.513876    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:02:18.390624 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:18.390587    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"]
Oct 30 21:02:18.498999 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:18.498966    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"]
Oct 30 21:02:18.799254 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:18.799220    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:18.974975 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:18.974942    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:23.391176 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.391101    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8" podUID="e16c027d-c495-4c3d-b5c5-b0a51016d26d" containerName="payload-processing" containerID="cri-o://6da7a489c51259305aee47787f7e43e808a40d4502aa9b459b51cb75479c7b49" gracePeriod=25
Oct 30 21:02:23.499439 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.499308    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67" podUID="fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" containerName="payload-pre-processing" containerID="cri-o://ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b" gracePeriod=25
Oct 30 21:02:23.588218 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.588186    5002 generic.go:356] "Generic (PLEG): container finished" podID="e16c027d-c495-4c3d-b5c5-b0a51016d26d" containerID="6da7a489c51259305aee47787f7e43e808a40d4502aa9b459b51cb75479c7b49" exitCode=0
Oct 30 21:02:23.588387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.588217    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8" event={"ID":"e16c027d-c495-4c3d-b5c5-b0a51016d26d","Type":"ContainerDied","Data":"6da7a489c51259305aee47787f7e43e808a40d4502aa9b459b51cb75479c7b49"}
Oct 30 21:02:23.593220 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.593185    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:23.700896 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.700096    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 21:02:23.782991 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.782965    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 21:02:23.813129 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.813098    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16c027d-c495-4c3d-b5c5-b0a51016d26d-kube-api-access-252kz\" (UniqueName: \"kubernetes.io/projected/e16c027d-c495-4c3d-b5c5-b0a51016d26d-kube-api-access-252kz\") pod \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") "
Oct 30 21:02:23.813309 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.813156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16c027d-c495-4c3d-b5c5-b0a51016d26d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16c027d-c495-4c3d-b5c5-b0a51016d26d-plugins-config-volume\") pod \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\" (UID: \"e16c027d-c495-4c3d-b5c5-b0a51016d26d\") "
Oct 30 21:02:23.813609 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.813573    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16c027d-c495-4c3d-b5c5-b0a51016d26d-plugins-config-volume" pod "e16c027d-c495-4c3d-b5c5-b0a51016d26d" (UID: "e16c027d-c495-4c3d-b5c5-b0a51016d26d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:23.815437 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.815413    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16c027d-c495-4c3d-b5c5-b0a51016d26d-kube-api-access-252kz" pod "e16c027d-c495-4c3d-b5c5-b0a51016d26d" (UID: "e16c027d-c495-4c3d-b5c5-b0a51016d26d"). InnerVolumeSpecName "kube-api-access-252kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:23.914261 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.914230    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-plugins-config-volume\") pod \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") "
Oct 30 21:02:23.914458 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.914384    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-kube-api-access-946fh\" (UniqueName: \"kubernetes.io/projected/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-kube-api-access-946fh\") pod \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\" (UID: \"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b\") "
Oct 30 21:02:23.914755 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.914731    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-plugins-config-volume" pod "fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" (UID: "fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:23.914867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.914850    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252kz\" (UniqueName: \"kubernetes.io/projected/e16c027d-c495-4c3d-b5c5-b0a51016d26d-kube-api-access-252kz\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:23.914915 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.914871    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16c027d-c495-4c3d-b5c5-b0a51016d26d-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:23.914915 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.914880    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:23.916894 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:23.916866    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-kube-api-access-946fh" pod "fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" (UID: "fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b"). InnerVolumeSpecName "kube-api-access-946fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:24.016008 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.015971    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946fh\" (UniqueName: \"kubernetes.io/projected/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b-kube-api-access-946fh\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:24.597465 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.597432    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"
Oct 30 21:02:24.597465 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.597448    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8" event={"ID":"e16c027d-c495-4c3d-b5c5-b0a51016d26d","Type":"ContainerDied","Data":"aab68a2df550782a2b6f223972723d034e9ac4d87a6879e4931828edf25be93d"}
Oct 30 21:02:24.598068 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.597521    5002 scope.go:122] "RemoveContainer" containerID="6da7a489c51259305aee47787f7e43e808a40d4502aa9b459b51cb75479c7b49"
Oct 30 21:02:24.599162 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.599140    5002 generic.go:356] "Generic (PLEG): container finished" podID="fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" containerID="ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b" exitCode=0
Oct 30 21:02:24.599254 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.599175    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67" event={"ID":"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b","Type":"ContainerDied","Data":"ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b"}
Oct 30 21:02:24.599254 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.599181    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"
Oct 30 21:02:24.599254 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.599212    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67" event={"ID":"fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b","Type":"ContainerDied","Data":"65e3db0193c491403276cf919f8270a6c7d958e3243a3a8da5f09f38a295670b"}
Oct 30 21:02:24.607971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.607944    5002 scope.go:122] "RemoveContainer" containerID="ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b"
Oct 30 21:02:24.617636 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.617612    5002 scope.go:122] "RemoveContainer" containerID="ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b"
Oct 30 21:02:24.617987 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:02:24.617965    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b\": container with ID starting with ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b not found: ID does not exist" containerID="ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b"
Oct 30 21:02:24.618055 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.618001    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b"} err="failed to get container status \"ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b\": rpc error: code = NotFound desc = could not find container \"ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b\": container with ID starting with ae42d95202df2bdf02499f39459d0ae7e934fc575d3142d209f4d1f1f5753a6b not found: ID does not exist"
Oct 30 21:02:24.770775 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.770725    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"]
Oct 30 21:02:24.820312 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.820270    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8"]
Oct 30 21:02:24.879546 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.879502    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"]
Oct 30 21:02:24.887333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:24.887297    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67"]
Oct 30 21:02:25.058508 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:25.058398    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:02:25.967490 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:25.967450    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16c027d-c495-4c3d-b5c5-b0a51016d26d" path="/var/lib/kubelet/pods/e16c027d-c495-4c3d-b5c5-b0a51016d26d/volumes"
Oct 30 21:02:25.967971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:25.967955    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" path="/var/lib/kubelet/pods/fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b/volumes"
Oct 30 21:02:26.785787 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:26.785752    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:28.292160 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:28.292126    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:32.998863 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:32.998821    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:37.116494 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:02:37.116450    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:02:37.715209 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:37.715172    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:38.687893 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:38.687856    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:42.488023 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:42.487984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:45.376025 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:45.375985    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:45.555221 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:45.555185    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:47.295596 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:47.295514    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:51.997677 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:51.997596    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:52.712697 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:52.712663    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:53.154602 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:53.154510    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:54.028822 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:54.028790    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:54.341727 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:54.341641    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:56.988372 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:56.988340    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:58.660019 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:58.659984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:58.847191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:58.847157    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:59.026977 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:02:59.026938    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:01.875579 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:01.875518    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:02.106223 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.106193    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:02.280947 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.280908    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:02.420719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.420682    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"]
Oct 30 21:03:02.421954 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.421925    5002 cpu_manager.go:414] "containerMap: removing container" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.421958    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.421979    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e16c027d-c495-4c3d-b5c5-b0a51016d26d" containerName="payload-processing"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.421987    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16c027d-c495-4c3d-b5c5-b0a51016d26d" containerName="payload-processing"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422017    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422025    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422057    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" containerName="payload-pre-processing"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422064    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" containerName="payload-pre-processing"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422095    5002 cpu_manager.go:414] "containerMap: removing container" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy"
Oct 30 21:03:02.422105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422103    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422123    5002 cpu_manager.go:414] "containerMap: removing container" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422130    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422289    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422304    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0ffbde-0875-4393-8f9b-2d4d4b9d9c5b" containerName="payload-pre-processing"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422317    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="88071f20-7d08-4bda-a6c6-65eccb3ab8e7" containerName="istio-proxy"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422327    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4d728a-78c0-405c-9907-6df33d6bd997" containerName="istio-proxy"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422337    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="911ca693-092e-40db-8487-3ec2f13bc8e2" containerName="cleanup"
Oct 30 21:03:02.422397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.422349    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16c027d-c495-4c3d-b5c5-b0a51016d26d" containerName="payload-processing"
Oct 30 21:03:02.427653 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.427627    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.455844 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.455816    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"]
Oct 30 21:03:02.481204 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.481154    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a064b4\""
Oct 30 21:03:02.481390 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.481282    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15d64ca-0058-43ef-90a2-b879e816a641-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.481390 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.481365    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jfk\" (UniqueName: \"kubernetes.io/projected/d15d64ca-0058-43ef-90a2-b879e816a641-kube-api-access-22jfk\") pod \"payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.481702 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.481680    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a064b4-dockercfg-tx84w\""
Oct 30 21:03:02.526900 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.526864    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:03:02.564258 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.564172    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"]
Oct 30 21:03:02.568271 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.568244    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:02.583194 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.583166    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15d64ca-0058-43ef-90a2-b879e816a641-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.583331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.583236    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jfk\" (UniqueName: \"kubernetes.io/projected/d15d64ca-0058-43ef-90a2-b879e816a641-kube-api-access-22jfk\") pod \"payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.584188 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.584160    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15d64ca-0058-43ef-90a2-b879e816a641-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.626300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.626265    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"]
Oct 30 21:03:02.642345 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.642312    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jfk\" (UniqueName: \"kubernetes.io/projected/d15d64ca-0058-43ef-90a2-b879e816a641-kube-api-access-22jfk\") pod \"payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.684142 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.684108    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf2sg\" (UniqueName: \"kubernetes.io/projected/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-kube-api-access-qf2sg\") pod \"payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:02.684364 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.684320    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:02.742880 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.742835    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:02.785479 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.785420    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:02.785720 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.785525    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf2sg\" (UniqueName: \"kubernetes.io/projected/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-kube-api-access-qf2sg\") pod \"payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:02.786244 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.786213    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:02.843711 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.843643    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf2sg\" (UniqueName: \"kubernetes.io/projected/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-kube-api-access-qf2sg\") pod \"payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:02.899565 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:02.897358    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:03.129117 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:03:03.129075    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15d64ca_0058_43ef_90a2_b879e816a641.slice/crio-0913252ff9f4c7775ac3294b0097cec7c8bbdee08cef10489fc05e8e021eab13 WatchSource:0}: Error finding container 0913252ff9f4c7775ac3294b0097cec7c8bbdee08cef10489fc05e8e021eab13: Status 404 returned error can't find the container with id 0913252ff9f4c7775ac3294b0097cec7c8bbdee08cef10489fc05e8e021eab13
Oct 30 21:03:03.130840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.130815    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"]
Oct 30 21:03:03.352355 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.352329    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"]
Oct 30 21:03:03.353220 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:03:03.353196    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a0ad7b_a3c5_496f_b75d_b295d8733fd6.slice/crio-91ee9130acf58d2fd44837c4b1bf38884d44a6ff91edfd4a0bfbef70d8f195ac WatchSource:0}: Error finding container 91ee9130acf58d2fd44837c4b1bf38884d44a6ff91edfd4a0bfbef70d8f195ac: Status 404 returned error can't find the container with id 91ee9130acf58d2fd44837c4b1bf38884d44a6ff91edfd4a0bfbef70d8f195ac
Oct 30 21:03:03.915663 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.915626    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" event={"ID":"41a0ad7b-a3c5-496f-b75d-b295d8733fd6","Type":"ContainerStarted","Data":"e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d"}
Oct 30 21:03:03.916108 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.915686    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" event={"ID":"41a0ad7b-a3c5-496f-b75d-b295d8733fd6","Type":"ContainerStarted","Data":"91ee9130acf58d2fd44837c4b1bf38884d44a6ff91edfd4a0bfbef70d8f195ac"}
Oct 30 21:03:03.916108 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.915717    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:03.917549 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.917511    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" event={"ID":"d15d64ca-0058-43ef-90a2-b879e816a641","Type":"ContainerStarted","Data":"54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7"}
Oct 30 21:03:03.917687 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.917591    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:03.917687 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:03.917618    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" event={"ID":"d15d64ca-0058-43ef-90a2-b879e816a641","Type":"ContainerStarted","Data":"0913252ff9f4c7775ac3294b0097cec7c8bbdee08cef10489fc05e8e021eab13"}
Oct 30 21:03:04.032006 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:04.031957    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" podStartSLOduration=2.031942586 podStartE2EDuration="2.031942586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:04.030186115 +0000 UTC m=+2610.179237102" watchObservedRunningTime="2026-10-30 21:03:04.031942586 +0000 UTC m=+2610.180993574"
Oct 30 21:03:04.152439 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:04.152373    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" podStartSLOduration=2.152344873 podStartE2EDuration="2.152344873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:04.150777977 +0000 UTC m=+2610.299829012" watchObservedRunningTime="2026-10-30 21:03:04.152344873 +0000 UTC m=+2610.301395864"
Oct 30 21:03:06.159865 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.159822    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"]
Oct 30 21:03:06.160428 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.160254    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt" podUID="6dcf1e60-ab9b-414a-a360-820275adfbff" containerName="maas-api" containerID="cri-o://a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c" gracePeriod=30
Oct 30 21:03:06.297285 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.297246    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:06.450768 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.450740    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 21:03:06.567819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.567174    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls\") pod \"6dcf1e60-ab9b-414a-a360-820275adfbff\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") "
Oct 30 21:03:06.567819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.567292    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcf1e60-ab9b-414a-a360-820275adfbff-kube-api-access-d4cwg\" (UniqueName: \"kubernetes.io/projected/6dcf1e60-ab9b-414a-a360-820275adfbff-kube-api-access-d4cwg\") pod \"6dcf1e60-ab9b-414a-a360-820275adfbff\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") "
Oct 30 21:03:06.567819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.567407    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dcf1e60-ab9b-414a-a360-820275adfbff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf1e60-ab9b-414a-a360-820275adfbff-openshift-service-ca\") pod \"6dcf1e60-ab9b-414a-a360-820275adfbff\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") "
Oct 30 21:03:06.567819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.567457    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-metrics-certs\") pod \"6dcf1e60-ab9b-414a-a360-820275adfbff\" (UID: \"6dcf1e60-ab9b-414a-a360-820275adfbff\") "
Oct 30 21:03:06.575205 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.572059    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-metrics-certs" pod "6dcf1e60-ab9b-414a-a360-820275adfbff" (UID: "6dcf1e60-ab9b-414a-a360-820275adfbff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:06.575205 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.573934    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcf1e60-ab9b-414a-a360-820275adfbff-openshift-service-ca" pod "6dcf1e60-ab9b-414a-a360-820275adfbff" (UID: "6dcf1e60-ab9b-414a-a360-820275adfbff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:06.577288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.576598    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls" pod "6dcf1e60-ab9b-414a-a360-820275adfbff" (UID: "6dcf1e60-ab9b-414a-a360-820275adfbff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:06.580123 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.580081    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcf1e60-ab9b-414a-a360-820275adfbff-kube-api-access-d4cwg" pod "6dcf1e60-ab9b-414a-a360-820275adfbff" (UID: "6dcf1e60-ab9b-414a-a360-820275adfbff"). InnerVolumeSpecName "kube-api-access-d4cwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:06.669584 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.669471    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:06.669584 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.669511    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4cwg\" (UniqueName: \"kubernetes.io/projected/6dcf1e60-ab9b-414a-a360-820275adfbff-kube-api-access-d4cwg\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:06.669584 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.669525    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf1e60-ab9b-414a-a360-820275adfbff-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:06.669584 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.669557    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dcf1e60-ab9b-414a-a360-820275adfbff-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:06.943289 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.943190    5002 generic.go:356] "Generic (PLEG): container finished" podID="6dcf1e60-ab9b-414a-a360-820275adfbff" containerID="a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c" exitCode=0
Oct 30 21:03:06.943289 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.943242    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"
Oct 30 21:03:06.943289 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.943280    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt" event={"ID":"6dcf1e60-ab9b-414a-a360-820275adfbff","Type":"ContainerDied","Data":"a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c"}
Oct 30 21:03:06.943607 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.943339    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt" event={"ID":"6dcf1e60-ab9b-414a-a360-820275adfbff","Type":"ContainerDied","Data":"c955d439019d5b967f6caf3f25ed9b5932afb975e5846723a9a8f4e30db84ad4"}
Oct 30 21:03:06.943607 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.943366    5002 scope.go:122] "RemoveContainer" containerID="a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c"
Oct 30 21:03:06.956081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.956052    5002 scope.go:122] "RemoveContainer" containerID="a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c"
Oct 30 21:03:06.956397 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:06.956372    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c\": container with ID starting with a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c not found: ID does not exist" containerID="a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c"
Oct 30 21:03:06.956479 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:06.956409    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c"} err="failed to get container status \"a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c\": rpc error: code = NotFound desc = could not find container \"a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c\": container with ID starting with a2ebe154e8d7133bc3c1fd194857cdad1540abe8195a13c1415ad19cf758d19c not found: ID does not exist"
Oct 30 21:03:07.120627 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:07.115320    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"]
Oct 30 21:03:07.159692 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:07.159658    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:03:07.164617 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:07.164588    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-23491e-cf8b4f55b-fskgt"]
Oct 30 21:03:07.833109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:07.833073    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:07.967043 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:07.967008    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcf1e60-ab9b-414a-a360-820275adfbff" path="/var/lib/kubelet/pods/6dcf1e60-ab9b-414a-a360-820275adfbff/volumes"
Oct 30 21:03:08.035587 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:08.035525    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:08.218988 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:08.218953    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:08.732218 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:08.732181    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:12.981391 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:12.981342    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:14.657668 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:14.657638    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:14.904340 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:14.904302    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:14.928112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:14.928035    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:03:14.929300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:14.929276    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:03:15.099891 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:15.099858    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:15.520813 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:15.520785    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:18.011976 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:18.011932    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:20.426733 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:20.426698    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"]
Oct 30 21:03:20.474996 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:20.474968    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"]
Oct 30 21:03:20.622618 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:20.622576    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:23.011792 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:23.011756    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:27.631207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:27.631170    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:27.990792 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:27.990760    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:28.164958 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:28.164921    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:31.594989 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.594947    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"]
Oct 30 21:03:31.595414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.595209    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9" podUID="81548445-8347-4e1b-9ec9-4d87ccc5df70" containerName="maas-api" containerID="cri-o://51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9" gracePeriod=30
Oct 30 21:03:31.737106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.737074    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:31.882944 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.882921    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 21:03:31.952830 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.952805    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81548445-8347-4e1b-9ec9-4d87ccc5df70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81548445-8347-4e1b-9ec9-4d87ccc5df70-openshift-service-ca\") pod \"81548445-8347-4e1b-9ec9-4d87ccc5df70\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") "
Oct 30 21:03:31.952959 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.952844    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-metrics-certs\") pod \"81548445-8347-4e1b-9ec9-4d87ccc5df70\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") "
Oct 30 21:03:31.952959 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.952891    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81548445-8347-4e1b-9ec9-4d87ccc5df70-kube-api-access-c79cf\" (UniqueName: \"kubernetes.io/projected/81548445-8347-4e1b-9ec9-4d87ccc5df70-kube-api-access-c79cf\") pod \"81548445-8347-4e1b-9ec9-4d87ccc5df70\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") "
Oct 30 21:03:31.952959 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.952934    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls\") pod \"81548445-8347-4e1b-9ec9-4d87ccc5df70\" (UID: \"81548445-8347-4e1b-9ec9-4d87ccc5df70\") "
Oct 30 21:03:31.953152 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.953124    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81548445-8347-4e1b-9ec9-4d87ccc5df70-openshift-service-ca" pod "81548445-8347-4e1b-9ec9-4d87ccc5df70" (UID: "81548445-8347-4e1b-9ec9-4d87ccc5df70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:31.953351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.953333    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81548445-8347-4e1b-9ec9-4d87ccc5df70-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:31.955091 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.955063    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls" pod "81548445-8347-4e1b-9ec9-4d87ccc5df70" (UID: "81548445-8347-4e1b-9ec9-4d87ccc5df70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:31.955203 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.955120    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-metrics-certs" pod "81548445-8347-4e1b-9ec9-4d87ccc5df70" (UID: "81548445-8347-4e1b-9ec9-4d87ccc5df70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:31.955203 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:31.955122    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81548445-8347-4e1b-9ec9-4d87ccc5df70-kube-api-access-c79cf" pod "81548445-8347-4e1b-9ec9-4d87ccc5df70" (UID: "81548445-8347-4e1b-9ec9-4d87ccc5df70"). InnerVolumeSpecName "kube-api-access-c79cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:32.054586 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.054520    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:32.054586 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.054589    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79cf\" (UniqueName: \"kubernetes.io/projected/81548445-8347-4e1b-9ec9-4d87ccc5df70-kube-api-access-c79cf\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:32.054821 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.054607    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81548445-8347-4e1b-9ec9-4d87ccc5df70-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:32.132306 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.132227    5002 generic.go:356] "Generic (PLEG): container finished" podID="81548445-8347-4e1b-9ec9-4d87ccc5df70" containerID="51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9" exitCode=0
Oct 30 21:03:32.132306 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.132297    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"
Oct 30 21:03:32.132501 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.132314    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9" event={"ID":"81548445-8347-4e1b-9ec9-4d87ccc5df70","Type":"ContainerDied","Data":"51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9"}
Oct 30 21:03:32.132501 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.132381    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9" event={"ID":"81548445-8347-4e1b-9ec9-4d87ccc5df70","Type":"ContainerDied","Data":"2f49156b5c6bc6d964984ca78f8f43eb01ab92e98257df049918fc1a1313564c"}
Oct 30 21:03:32.132501 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.132409    5002 scope.go:122] "RemoveContainer" containerID="51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9"
Oct 30 21:03:32.145795 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.145772    5002 scope.go:122] "RemoveContainer" containerID="51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9"
Oct 30 21:03:32.146101 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:32.146076    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9\": container with ID starting with 51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9 not found: ID does not exist" containerID="51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9"
Oct 30 21:03:32.146179 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.146110    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9"} err="failed to get container status \"51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9\": rpc error: code = NotFound desc = could not find container \"51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9\": container with ID starting with 51fd606f0fcf22b298440171e13678cdd6276b7575024dbbb2069f59f96e79a9 not found: ID does not exist"
Oct 30 21:03:32.298551 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.298509    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"]
Oct 30 21:03:32.301299 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:32.301279    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-92f229-5554477548-x2dd9"]
Oct 30 21:03:33.276835 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:33.276804    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:33.523637 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:33.523608    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:33.699659 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:33.699574    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:33.966915 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:33.966888    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81548445-8347-4e1b-9ec9-4d87ccc5df70" path="/var/lib/kubelet/pods/81548445-8347-4e1b-9ec9-4d87ccc5df70/volumes"
Oct 30 21:03:34.200550 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:34.200513    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:35.448830 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.448726    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" containerID="cri-o://e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054" gracePeriod=45
Oct 30 21:03:35.495295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.495245    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main" containerID="cri-o://324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19" gracePeriod=45
Oct 30 21:03:35.779160 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.778840    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 21:03:35.800902 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.800877    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 21:03:35.895496 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895462    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-dshm\") pod \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") "
Oct 30 21:03:35.895709 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895520    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-home\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-home\") pod \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") "
Oct 30 21:03:35.895709 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895582    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tls-certs\" (UniqueName: \"kubernetes.io/secret/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tls-certs\") pod \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") "
Oct 30 21:03:35.895709 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895612    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-model-cache\") pod \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") "
Oct 30 21:03:35.895709 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895680    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-model-cache\") pod \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") "
Oct 30 21:03:35.895993 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895717    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tmp-dir\") pod \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") "
Oct 30 21:03:35.895993 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895833    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-kube-api-access-dbnfs\" (UniqueName: \"kubernetes.io/projected/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-kube-api-access-dbnfs\") pod \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") "
Oct 30 21:03:35.895993 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895872    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tmp-dir\") pod \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") "
Oct 30 21:03:35.895993 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.895922    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-home\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-home\") pod \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") "
Oct 30 21:03:35.896225 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896016    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-kube-api-access-ksp9h\" (UniqueName: \"kubernetes.io/projected/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-kube-api-access-ksp9h\") pod \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") "
Oct 30 21:03:35.896225 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896050    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-dshm\") pod \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\" (UID: \"79a2ec9a-b6e5-42fc-ad00-3cd0267db026\") "
Oct 30 21:03:35.896225 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896088    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tls-certs\" (UniqueName: \"kubernetes.io/secret/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tls-certs\") pod \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\" (UID: \"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019\") "
Oct 30 21:03:35.896390 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896361    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-model-cache" pod "79a2ec9a-b6e5-42fc-ad00-3cd0267db026" (UID: "79a2ec9a-b6e5-42fc-ad00-3cd0267db026"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896586    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tmp-dir" pod "79a2ec9a-b6e5-42fc-ad00-3cd0267db026" (UID: "79a2ec9a-b6e5-42fc-ad00-3cd0267db026"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896624    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-home" pod "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" (UID: "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896678    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-model-cache" pod "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" (UID: "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896786    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-home" pod "79a2ec9a-b6e5-42fc-ad00-3cd0267db026" (UID: "79a2ec9a-b6e5-42fc-ad00-3cd0267db026"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896884    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tmp-dir" pod "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" (UID: "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896974    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.896993    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.897006    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.897020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.897024    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.897573 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.897039    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.897573 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.897054    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.903443 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.903420    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-dshm" pod "79a2ec9a-b6e5-42fc-ad00-3cd0267db026" (UID: "79a2ec9a-b6e5-42fc-ad00-3cd0267db026"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.905188 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.905137    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-kube-api-access-ksp9h" pod "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" (UID: "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019"). InnerVolumeSpecName "kube-api-access-ksp9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:35.906465 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.906431    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-kube-api-access-dbnfs" pod "79a2ec9a-b6e5-42fc-ad00-3cd0267db026" (UID: "79a2ec9a-b6e5-42fc-ad00-3cd0267db026"). InnerVolumeSpecName "kube-api-access-dbnfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:35.907783 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.907764    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-dshm" pod "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" (UID: "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.909075 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.909050    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tls-certs" pod "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" (UID: "80b26da6-9ac0-4f6e-a540-cb0dc4a0d019"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:35.909319 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.909299    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tls-certs" pod "79a2ec9a-b6e5-42fc-ad00-3cd0267db026" (UID: "79a2ec9a-b6e5-42fc-ad00-3cd0267db026"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:35.998142 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.998115    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksp9h\" (UniqueName: \"kubernetes.io/projected/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-kube-api-access-ksp9h\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.998142 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.998140    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.998282 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.998150    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.998282 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.998159    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.998282 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.998167    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.998282 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:35.998175    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbnfs\" (UniqueName: \"kubernetes.io/projected/79a2ec9a-b6e5-42fc-ad00-3cd0267db026-kube-api-access-dbnfs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:36.164508 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.164460    5002 generic.go:356] "Generic (PLEG): container finished" podID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerID="324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19" exitCode=0
Oct 30 21:03:36.164508 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.164505    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"
Oct 30 21:03:36.164812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.164566    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" event={"ID":"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019","Type":"ContainerDied","Data":"324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19"}
Oct 30 21:03:36.164812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.164606    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7" event={"ID":"80b26da6-9ac0-4f6e-a540-cb0dc4a0d019","Type":"ContainerDied","Data":"62e4de510e0daab2ddede6fecabe99f6b225525a401ee7bb2b724a3310d88a26"}
Oct 30 21:03:36.164812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.164626    5002 scope.go:122] "RemoveContainer" containerID="324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19"
Oct 30 21:03:36.166053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.165946    5002 generic.go:356] "Generic (PLEG): container finished" podID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerID="e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054" exitCode=0
Oct 30 21:03:36.166053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.165975    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" event={"ID":"79a2ec9a-b6e5-42fc-ad00-3cd0267db026","Type":"ContainerDied","Data":"e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054"}
Oct 30 21:03:36.166053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.166037    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" event={"ID":"79a2ec9a-b6e5-42fc-ad00-3cd0267db026","Type":"ContainerDied","Data":"1aa5a0532a570fe7410a229c044960333deb4c894edaa27c6e41bdf9c4433a29"}
Oct 30 21:03:36.166053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.166001    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"
Oct 30 21:03:36.174661 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.174634    5002 scope.go:122] "RemoveContainer" containerID="324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19"
Oct 30 21:03:36.174938 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:36.174921    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19\": container with ID starting with 324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19 not found: ID does not exist" containerID="324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19"
Oct 30 21:03:36.174992 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.174948    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19"} err="failed to get container status \"324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19\": rpc error: code = NotFound desc = could not find container \"324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19\": container with ID starting with 324b9cd02f988bef1332e462e1e8bee3c5c064b583609b20a23c0827f1f6eb19 not found: ID does not exist"
Oct 30 21:03:36.174992 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.174963    5002 scope.go:122] "RemoveContainer" containerID="e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054"
Oct 30 21:03:36.183754 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.183740    5002 scope.go:122] "RemoveContainer" containerID="e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054"
Oct 30 21:03:36.183992 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:36.183975    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054\": container with ID starting with e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054 not found: ID does not exist" containerID="e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054"
Oct 30 21:03:36.184053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.184003    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054"} err="failed to get container status \"e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054\": rpc error: code = NotFound desc = could not find container \"e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054\": container with ID starting with e31699b6971ba8b3964e22ecf9af7dfb62ce87518c8581aeb0048df845223054 not found: ID does not exist"
Oct 30 21:03:36.332703 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.332629    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"]
Oct 30 21:03:36.338852 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.338829    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/premium-w2-df031b-kserve-66d54fbd4d-5mpv7"]
Oct 30 21:03:36.449486 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.449458    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"]
Oct 30 21:03:36.453850 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.453826    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5"]
Oct 30 21:03:36.685669 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:36.685581    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-w2-df031b-kserve-5bf4598479-8m9v5" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:37.115593 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:37.115558    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:03:37.966989 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:37.966953    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" path="/var/lib/kubelet/pods/79a2ec9a-b6e5-42fc-ad00-3cd0267db026/volumes"
Oct 30 21:03:37.967375 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:37.967342    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" path="/var/lib/kubelet/pods/80b26da6-9ac0-4f6e-a540-cb0dc4a0d019/volumes"
Oct 30 21:03:38.227392 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227323    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"]
Oct 30 21:03:38.227871 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227856    5002 cpu_manager.go:414] "containerMap: removing container" podUID="81548445-8347-4e1b-9ec9-4d87ccc5df70" containerName="maas-api"
Oct 30 21:03:38.227935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227873    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81548445-8347-4e1b-9ec9-4d87ccc5df70" containerName="maas-api"
Oct 30 21:03:38.227935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227895    5002 cpu_manager.go:414] "containerMap: removing container" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main"
Oct 30 21:03:38.227935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227900    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main"
Oct 30 21:03:38.227935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227908    5002 cpu_manager.go:414] "containerMap: removing container" podUID="6dcf1e60-ab9b-414a-a360-820275adfbff" containerName="maas-api"
Oct 30 21:03:38.227935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227913    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcf1e60-ab9b-414a-a360-820275adfbff" containerName="maas-api"
Oct 30 21:03:38.227935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227936    5002 cpu_manager.go:414] "containerMap: removing container" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main"
Oct 30 21:03:38.228112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.227941    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main"
Oct 30 21:03:38.228112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.228023    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a2ec9a-b6e5-42fc-ad00-3cd0267db026" containerName="main"
Oct 30 21:03:38.228112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.228035    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="81548445-8347-4e1b-9ec9-4d87ccc5df70" containerName="maas-api"
Oct 30 21:03:38.228112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.228044    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b26da6-9ac0-4f6e-a540-cb0dc4a0d019" containerName="main"
Oct 30 21:03:38.228112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.228050    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcf1e60-ab9b-414a-a360-820275adfbff" containerName="maas-api"
Oct 30 21:03:38.232736 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.232718    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.289517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.289493    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"]
Oct 30 21:03:38.321169 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.321125    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8945\" (UniqueName: \"kubernetes.io/projected/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-kube-api-access-g8945\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.321313 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.321205    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-metrics-certs\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.321389 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.321364    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.321467 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.321447    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.336719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.336698    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a064b4-serving-cert\""
Oct 30 21:03:38.422258 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.422217    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.422416 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.422275    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.422416 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.422320    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8945\" (UniqueName: \"kubernetes.io/projected/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-kube-api-access-g8945\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.422416 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:38.422343    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-serving-cert: secret "maas-api-e2e-inf-b-a064b4-serving-cert" not found
Oct 30 21:03:38.422416 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.422371    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-metrics-certs\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.422416 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:38.422405    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls podName:c98bdadb-0093-49eb-b21c-92b0c19fb4ac nodeName:}" failed. No retries permitted until 2026-10-30 21:03:38.922390766 +0000 UTC m=+2645.071441747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls") pod "maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk" (UID: "c98bdadb-0093-49eb-b21c-92b0c19fb4ac") : secret "maas-api-e2e-inf-b-a064b4-serving-cert" not found
Oct 30 21:03:38.423010 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.422994    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.425526 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.425490    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-metrics-certs\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.454743 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.454708    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:38.481723 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.481659    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8945\" (UniqueName: \"kubernetes.io/projected/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-kube-api-access-g8945\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.637791 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.637747    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:38.818376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.818293    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:38.927296 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.927260    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:38.929989 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:38.929961    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:39.017266 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:39.017239    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:39.146207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:39.146141    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:39.524905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:39.524864    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"]
Oct 30 21:03:39.525915 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:03:39.525891    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98bdadb_0093_49eb_b21c_92b0c19fb4ac.slice/crio-66465fc4578fe2ef4d03e48c715d571e5827b3ccdc0aa6ee47558c8165fdb559 WatchSource:0}: Error finding container 66465fc4578fe2ef4d03e48c715d571e5827b3ccdc0aa6ee47558c8165fdb559: Status 404 returned error can't find the container with id 66465fc4578fe2ef4d03e48c715d571e5827b3ccdc0aa6ee47558c8165fdb559
Oct 30 21:03:40.188410 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:40.188378    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:40.198235 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:40.198196    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk" event={"ID":"c98bdadb-0093-49eb-b21c-92b0c19fb4ac","Type":"ContainerStarted","Data":"66465fc4578fe2ef4d03e48c715d571e5827b3ccdc0aa6ee47558c8165fdb559"}
Oct 30 21:03:40.363761 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:40.363729    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:40.492411 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:40.492383    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"]
Oct 30 21:03:40.492614 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:40.492594    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy" containerID="cri-o://39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56" gracePeriod=30
Oct 30 21:03:40.555504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:40.555460    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:41.015403 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:41.015349    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:41.153412 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:41.153363    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"]
Oct 30 21:03:41.206464 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:41.206434    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk" event={"ID":"c98bdadb-0093-49eb-b21c-92b0c19fb4ac","Type":"ContainerStarted","Data":"8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f"}
Oct 30 21:03:41.206858 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:41.206563    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:41.235863 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:41.235833    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:41.390295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:41.390190    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk" podStartSLOduration=2.495084882 podStartE2EDuration="3.390175146s" totalImagesPullingTime="895.090264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:41.387156237 +0000 UTC m=+2647.536207231" watchObservedRunningTime="2026-10-30 21:03:41.390175146 +0000 UTC m=+2647.539226140"
Oct 30 21:03:43.737562 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:43.736947    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:45.777792 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.777760    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 21:03:45.907178 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907147    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istiod-ca-cert\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907311 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907219    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-kube-api-access-4wzjt\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-kube-api-access-4wzjt\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907369    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-certs\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907438 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907428    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-socket\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907455    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-credential-socket\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907467    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istiod-ca-cert" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:45.907504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907483    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-data\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907573    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-podinfo\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907627    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-envoy\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.907700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907685    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-socket" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:45.907837 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907729    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-certs" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:45.907837 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907780    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-credential-socket" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:45.907837 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.907784    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-token\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-token\") pod \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\" (UID: \"efe31d77-e37f-41b1-87fb-8a3eb5374f40\") "
Oct 30 21:03:45.908148 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.908126    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.908231 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.908153    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.908231 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.908165    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.908231 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.908178    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.908591 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.908290    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-data" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:45.909685 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.909655    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-kube-api-access-4wzjt" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "kube-api-access-4wzjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:45.909941 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.909914    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-envoy" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:45.910504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.910477    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-podinfo" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:45.910633 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:45.910559    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-token" pod "efe31d77-e37f-41b1-87fb-8a3eb5374f40" (UID: "efe31d77-e37f-41b1-87fb-8a3eb5374f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:46.009483 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.009409    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:46.009483 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.009446    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzjt\" (UniqueName: \"kubernetes.io/projected/efe31d77-e37f-41b1-87fb-8a3eb5374f40-kube-api-access-4wzjt\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:46.009483 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.009462    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:46.009697 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.009493    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:46.009697 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.009507    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe31d77-e37f-41b1-87fb-8a3eb5374f40-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:46.138386 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.138352    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:46.247198 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.247161    5002 generic.go:356] "Generic (PLEG): container finished" podID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerID="39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56" exitCode=0
Oct 30 21:03:46.247380 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.247218    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" event={"ID":"efe31d77-e37f-41b1-87fb-8a3eb5374f40","Type":"ContainerDied","Data":"39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56"}
Oct 30 21:03:46.247380 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.247214    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"
Oct 30 21:03:46.247380 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.247266    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj" event={"ID":"efe31d77-e37f-41b1-87fb-8a3eb5374f40","Type":"ContainerDied","Data":"4417e600659d1e22074149f9e19442bd4ff62e21d005ab0f0301d18d23070f9e"}
Oct 30 21:03:46.247380 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.247295    5002 scope.go:122] "RemoveContainer" containerID="39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56"
Oct 30 21:03:46.257718 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.257697    5002 scope.go:122] "RemoveContainer" containerID="70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a"
Oct 30 21:03:46.266469 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.266451    5002 scope.go:122] "RemoveContainer" containerID="39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56"
Oct 30 21:03:46.266746 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:46.266725    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56\": container with ID starting with 39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56 not found: ID does not exist" containerID="39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56"
Oct 30 21:03:46.266811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.266754    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56"} err="failed to get container status \"39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56\": rpc error: code = NotFound desc = could not find container \"39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56\": container with ID starting with 39be32bf4c1556e1c7b2ac7eb6361a5974eae0d09504fbec656988a693ff0a56 not found: ID does not exist"
Oct 30 21:03:46.266811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.266773    5002 scope.go:122] "RemoveContainer" containerID="70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a"
Oct 30 21:03:46.267029 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:03:46.267010    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a\": container with ID starting with 70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a not found: ID does not exist" containerID="70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a"
Oct 30 21:03:46.267076 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.267036    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a"} err="failed to get container status \"70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a\": rpc error: code = NotFound desc = could not find container \"70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a\": container with ID starting with 70bef4ff4fee9cc261ad1c51415be792b7985a910e94e0bee00ff3d955fac89a not found: ID does not exist"
Oct 30 21:03:46.416757 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.416728    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"]
Oct 30 21:03:46.465211 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:46.465184    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj"]
Oct 30 21:03:47.222378 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:47.222345    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:03:47.973095 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:47.973052    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" path="/var/lib/kubelet/pods/efe31d77-e37f-41b1-87fb-8a3eb5374f40/volumes"
Oct 30 21:03:48.558152 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:48.558117    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:03:49.323937 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:49.323904    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:53.103841 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:53.103806    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"]
Oct 30 21:03:53.523199 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:53.522960    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:53.712131 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:53.712097    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:54.062653 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:54.062616    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:56.174951 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.174910    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42" podUID="4cbdc811-65cc-4aba-8be5-5a0150bd4402" containerName="main" containerID="cri-o://056b63b37ef65911d932f270b6a29353a72fda7352e0f222c9c5d1a3c3350865" gracePeriod=45
Oct 30 21:03:56.273680 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.273643    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42" podUID="4cbdc811-65cc-4aba-8be5-5a0150bd4402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": dial tcp 10.132.0.156:8000: connect: connection refused"
Oct 30 21:03:56.318732 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.318696    5002 generic.go:356] "Generic (PLEG): container finished" podID="4cbdc811-65cc-4aba-8be5-5a0150bd4402" containerID="056b63b37ef65911d932f270b6a29353a72fda7352e0f222c9c5d1a3c3350865" exitCode=0
Oct 30 21:03:56.318907 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.318763    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42" event={"ID":"4cbdc811-65cc-4aba-8be5-5a0150bd4402","Type":"ContainerDied","Data":"056b63b37ef65911d932f270b6a29353a72fda7352e0f222c9c5d1a3c3350865"}
Oct 30 21:03:56.431877 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.431816    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 21:03:56.540715 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.540672    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-home\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-home\") pod \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") "
Oct 30 21:03:56.540899 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.540770    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tls-certs\") pod \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") "
Oct 30 21:03:56.540899 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.540805    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tmp-dir\") pod \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") "
Oct 30 21:03:56.540899 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.540876    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbdc811-65cc-4aba-8be5-5a0150bd4402-kube-api-access-5fq9x\" (UniqueName: \"kubernetes.io/projected/4cbdc811-65cc-4aba-8be5-5a0150bd4402-kube-api-access-5fq9x\") pod \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") "
Oct 30 21:03:56.541085 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.540902    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-model-cache\") pod \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") "
Oct 30 21:03:56.541085 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.540927    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-dshm\") pod \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\" (UID: \"4cbdc811-65cc-4aba-8be5-5a0150bd4402\") "
Oct 30 21:03:56.541192 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.541124    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tmp-dir" pod "4cbdc811-65cc-4aba-8be5-5a0150bd4402" (UID: "4cbdc811-65cc-4aba-8be5-5a0150bd4402"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:56.541252 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.541207    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-home" pod "4cbdc811-65cc-4aba-8be5-5a0150bd4402" (UID: "4cbdc811-65cc-4aba-8be5-5a0150bd4402"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:56.541372 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.541332    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-model-cache" pod "4cbdc811-65cc-4aba-8be5-5a0150bd4402" (UID: "4cbdc811-65cc-4aba-8be5-5a0150bd4402"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:56.541544 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.541480    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:56.541544 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.541496    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:56.541544 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.541504    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:56.543432 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.543411    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-dshm" pod "4cbdc811-65cc-4aba-8be5-5a0150bd4402" (UID: "4cbdc811-65cc-4aba-8be5-5a0150bd4402"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:56.543673 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.543644    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbdc811-65cc-4aba-8be5-5a0150bd4402-kube-api-access-5fq9x" pod "4cbdc811-65cc-4aba-8be5-5a0150bd4402" (UID: "4cbdc811-65cc-4aba-8be5-5a0150bd4402"). InnerVolumeSpecName "kube-api-access-5fq9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:56.543786 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.543704    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tls-certs" pod "4cbdc811-65cc-4aba-8be5-5a0150bd4402" (UID: "4cbdc811-65cc-4aba-8be5-5a0150bd4402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:56.642876 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.642841    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbdc811-65cc-4aba-8be5-5a0150bd4402-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:56.642876 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.642876    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fq9x\" (UniqueName: \"kubernetes.io/projected/4cbdc811-65cc-4aba-8be5-5a0150bd4402-kube-api-access-5fq9x\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:56.643109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:56.642886    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbdc811-65cc-4aba-8be5-5a0150bd4402-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:57.326299 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.326268    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42" event={"ID":"4cbdc811-65cc-4aba-8be5-5a0150bd4402","Type":"ContainerDied","Data":"d980d04f90a93117494f7b0d963ec5d10e6cfd3304efe130650b5347d0984a22"}
Oct 30 21:03:57.326299 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.326284    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"
Oct 30 21:03:57.326830 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.326316    5002 scope.go:122] "RemoveContainer" containerID="056b63b37ef65911d932f270b6a29353a72fda7352e0f222c9c5d1a3c3350865"
Oct 30 21:03:57.495566 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.495517    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"]
Oct 30 21:03:57.546353 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.546328    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bcef0e/test-model-bcef0e-kserve-8db8cb6b9-8rq42"]
Oct 30 21:03:57.790344 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.790301    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:57.966221 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.966191    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbdc811-65cc-4aba-8be5-5a0150bd4402" path="/var/lib/kubelet/pods/4cbdc811-65cc-4aba-8be5-5a0150bd4402/volumes"
Oct 30 21:03:57.967470 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:57.967443    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:58.104772 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.104707    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6" podUID="1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" containerName="payload-pre-processing" containerID="cri-o://86a0d22c93b9808eeb81e7885342a4c7e6e6dc350b876597d05d122f77a931e3" gracePeriod=25
Oct 30 21:03:58.145916 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.145891    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:58.337062 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.337027    5002 generic.go:356] "Generic (PLEG): container finished" podID="1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" containerID="86a0d22c93b9808eeb81e7885342a4c7e6e6dc350b876597d05d122f77a931e3" exitCode=0
Oct 30 21:03:58.337555 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.337102    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6" event={"ID":"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb","Type":"ContainerDied","Data":"86a0d22c93b9808eeb81e7885342a4c7e6e6dc350b876597d05d122f77a931e3"}
Oct 30 21:03:58.370112 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.370065    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 21:03:58.465029 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.465004    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-kube-api-access-bs7dq\" (UniqueName: \"kubernetes.io/projected/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-kube-api-access-bs7dq\") pod \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") "
Oct 30 21:03:58.465180 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.465090    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-plugins-config-volume\") pod \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\" (UID: \"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb\") "
Oct 30 21:03:58.465400 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.465382    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-plugins-config-volume" pod "1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" (UID: "1b7dc019-30bf-4aa9-a2f4-91699d26dfdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:58.467289 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.467271    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-kube-api-access-bs7dq" pod "1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" (UID: "1b7dc019-30bf-4aa9-a2f4-91699d26dfdb"). InnerVolumeSpecName "kube-api-access-bs7dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:58.566910 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.566882    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:58.566910 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.566907    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7dq\" (UniqueName: \"kubernetes.io/projected/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb-kube-api-access-bs7dq\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:58.867103 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:58.867070    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:59.345840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:59.345811    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"
Oct 30 21:03:59.346335 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:59.345849    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6" event={"ID":"1b7dc019-30bf-4aa9-a2f4-91699d26dfdb","Type":"ContainerDied","Data":"14395a9b1b39f971ac71217cd5d99fad03e5c2c9a012dd8a12edcf647ed3b65e"}
Oct 30 21:03:59.346335 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:59.345910    5002 scope.go:122] "RemoveContainer" containerID="86a0d22c93b9808eeb81e7885342a4c7e6e6dc350b876597d05d122f77a931e3"
Oct 30 21:03:59.512019 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:59.511984    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"]
Oct 30 21:03:59.562301 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:59.562274    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6"]
Oct 30 21:03:59.966084 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:03:59.966050    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" path="/var/lib/kubelet/pods/1b7dc019-30bf-4aa9-a2f4-91699d26dfdb/volumes"
Oct 30 21:04:03.547245 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:03.547201    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:03.714806 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:03.714739    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:07.116457 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:04:07.116410    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:04:08.179582 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:08.179512    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:12.882964 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:12.882930    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:15.144198 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:15.144166    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:17.876491 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:17.876452    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:18.099519 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:18.099480    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:18.281370 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:18.281329    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:23.165240 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:23.165195    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:23.335483 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:23.335448    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:24.485129 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:24.485090    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:24.674894 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:24.674859    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:25.220725 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:25.220685    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:25.541582 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:25.541474    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:04:27.025162 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:27.025075    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"]
Oct 30 21:04:27.135028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:27.134995    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"]
Oct 30 21:04:27.444470 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:27.444389    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:27.645725 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:27.645691    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:27.817468 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:27.817432    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:29.375234 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:29.375198    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:29.595324 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:29.595291    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:29.773582 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:29.773521    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:30.298729 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:30.298660    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:30.481480 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:30.481439    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:32.026109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.026020    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q" podUID="d900bc8d-4020-4ec0-8dbe-1dc23712633b" containerName="payload-processing" containerID="cri-o://b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793" gracePeriod=25
Oct 30 21:04:32.135830 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.135750    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" podUID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" containerName="payload-pre-processing" containerID="cri-o://c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7" gracePeriod=25
Oct 30 21:04:32.303641 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.303611    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 21:04:32.348110 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.348078    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d900bc8d-4020-4ec0-8dbe-1dc23712633b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900bc8d-4020-4ec0-8dbe-1dc23712633b-plugins-config-volume\") pod \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") "
Oct 30 21:04:32.348281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.348171    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d900bc8d-4020-4ec0-8dbe-1dc23712633b-kube-api-access-ld755\" (UniqueName: \"kubernetes.io/projected/d900bc8d-4020-4ec0-8dbe-1dc23712633b-kube-api-access-ld755\") pod \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\" (UID: \"d900bc8d-4020-4ec0-8dbe-1dc23712633b\") "
Oct 30 21:04:32.348437 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.348414    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d900bc8d-4020-4ec0-8dbe-1dc23712633b-plugins-config-volume" pod "d900bc8d-4020-4ec0-8dbe-1dc23712633b" (UID: "d900bc8d-4020-4ec0-8dbe-1dc23712633b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:32.350428 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.350406    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d900bc8d-4020-4ec0-8dbe-1dc23712633b-kube-api-access-ld755" pod "d900bc8d-4020-4ec0-8dbe-1dc23712633b" (UID: "d900bc8d-4020-4ec0-8dbe-1dc23712633b"). InnerVolumeSpecName "kube-api-access-ld755". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:32.450030 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.449993    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld755\" (UniqueName: \"kubernetes.io/projected/d900bc8d-4020-4ec0-8dbe-1dc23712633b-kube-api-access-ld755\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:32.450030 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.450025    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900bc8d-4020-4ec0-8dbe-1dc23712633b-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:32.562438 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.562357    5002 generic.go:356] "Generic (PLEG): container finished" podID="d900bc8d-4020-4ec0-8dbe-1dc23712633b" containerID="b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793" exitCode=0
Oct 30 21:04:32.562438 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.562410    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"
Oct 30 21:04:32.562668 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.562451    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q" event={"ID":"d900bc8d-4020-4ec0-8dbe-1dc23712633b","Type":"ContainerDied","Data":"b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793"}
Oct 30 21:04:32.562668 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.562509    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q" event={"ID":"d900bc8d-4020-4ec0-8dbe-1dc23712633b","Type":"ContainerDied","Data":"ebccd78423cff98cddaf7b77f6f1d0fb0419e75b40dbb8b53b7da84ef7aa1100"}
Oct 30 21:04:32.562668 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.562548    5002 scope.go:122] "RemoveContainer" containerID="b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793"
Oct 30 21:04:32.572884 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.572858    5002 scope.go:122] "RemoveContainer" containerID="b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793"
Oct 30 21:04:32.573166 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:04:32.573148    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793\": container with ID starting with b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793 not found: ID does not exist" containerID="b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793"
Oct 30 21:04:32.573206 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.573177    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793"} err="failed to get container status \"b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793\": rpc error: code = NotFound desc = could not find container \"b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793\": container with ID starting with b30f57f1786ceef097f1c02aa13fa248dbe6d6b88f0b222b6fe5bd584fa1f793 not found: ID does not exist"
Oct 30 21:04:32.727916 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.727868    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"]
Oct 30 21:04:32.731996 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:32.731965    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q"]
Oct 30 21:04:33.967366 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:33.967330    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d900bc8d-4020-4ec0-8dbe-1dc23712633b" path="/var/lib/kubelet/pods/d900bc8d-4020-4ec0-8dbe-1dc23712633b/volumes"
Oct 30 21:04:34.396136 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:34.396066    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:04:34.417645 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:34.417619    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:04:34.568999 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:34.568954    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:36.696982 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:36.696951    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:37.131167 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:04:37.131130    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:04:38.124735 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:38.124699    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:38.373173 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:38.373138    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" podUID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:04:39.350096 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:39.350042    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:39.523602 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:39.523565    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:44.193507 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:44.193468    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:48.373339 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:48.373300    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" podUID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:04:48.901253 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:48.901216    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:52.423936 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.423906    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 21:04:52.572999 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.572962    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-plugins-config-volume\") pod \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") "
Oct 30 21:04:52.573202 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.573137    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-kube-api-access-rstgs\" (UniqueName: \"kubernetes.io/projected/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-kube-api-access-rstgs\") pod \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\" (UID: \"8812bf72-cfe9-4e13-9196-96e2ab52b2fd\") "
Oct 30 21:04:52.573399 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.573374    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-plugins-config-volume" pod "8812bf72-cfe9-4e13-9196-96e2ab52b2fd" (UID: "8812bf72-cfe9-4e13-9196-96e2ab52b2fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:52.573614 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.573597    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:52.575485 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.575456    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-kube-api-access-rstgs" pod "8812bf72-cfe9-4e13-9196-96e2ab52b2fd" (UID: "8812bf72-cfe9-4e13-9196-96e2ab52b2fd"). InnerVolumeSpecName "kube-api-access-rstgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:52.674296 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.674219    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstgs\" (UniqueName: \"kubernetes.io/projected/8812bf72-cfe9-4e13-9196-96e2ab52b2fd-kube-api-access-rstgs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:52.692591 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.692558    5002 generic.go:356] "Generic (PLEG): container finished" podID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" containerID="c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7" exitCode=0
Oct 30 21:04:52.692719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.692605    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"
Oct 30 21:04:52.692719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.692633    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" event={"ID":"8812bf72-cfe9-4e13-9196-96e2ab52b2fd","Type":"ContainerDied","Data":"c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7"}
Oct 30 21:04:52.692719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.692692    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql" event={"ID":"8812bf72-cfe9-4e13-9196-96e2ab52b2fd","Type":"ContainerDied","Data":"36725ad2f1c47edf72340e9510686dec404f1dcdd885acfc51663f4be72b5c99"}
Oct 30 21:04:52.692719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.692714    5002 scope.go:122] "RemoveContainer" containerID="c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7"
Oct 30 21:04:52.703131 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.703116    5002 scope.go:122] "RemoveContainer" containerID="c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7"
Oct 30 21:04:52.703396 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:04:52.703375    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7\": container with ID starting with c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7 not found: ID does not exist" containerID="c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7"
Oct 30 21:04:52.703444 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.703407    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7"} err="failed to get container status \"c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7\": rpc error: code = NotFound desc = could not find container \"c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7\": container with ID starting with c3804831870d91a9bf1d37f15d5031b3a2b86a970ed92102a2c1fd517c5846d7 not found: ID does not exist"
Oct 30 21:04:52.860235 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.860177    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"]
Oct 30 21:04:52.863496 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:52.863472    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql"]
Oct 30 21:04:53.585880 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:53.585826    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:53.656522 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:53.656488    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:04:53.967295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:53.967262    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" path="/var/lib/kubelet/pods/8812bf72-cfe9-4e13-9196-96e2ab52b2fd/volumes"
Oct 30 21:04:54.494174 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:54.494134    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:54.676513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:54.676479    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:54.863776 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:54.863698    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:57.695387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:57.695352    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:58.585790 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.585752    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"]
Oct 30 21:04:58.586282 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586269    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d900bc8d-4020-4ec0-8dbe-1dc23712633b" containerName="payload-processing"
Oct 30 21:04:58.586331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586285    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d900bc8d-4020-4ec0-8dbe-1dc23712633b" containerName="payload-processing"
Oct 30 21:04:58.586331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586304    5002 cpu_manager.go:414] "containerMap: removing container" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy"
Oct 30 21:04:58.586331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586309    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy"
Oct 30 21:04:58.586331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586317    5002 cpu_manager.go:414] "containerMap: removing container" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy"
Oct 30 21:04:58.586331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586322    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586333    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" containerName="payload-pre-processing"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586338    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" containerName="payload-pre-processing"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586357    5002 cpu_manager.go:414] "containerMap: removing container" podUID="1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" containerName="payload-pre-processing"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586362    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" containerName="payload-pre-processing"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586368    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4cbdc811-65cc-4aba-8be5-5a0150bd4402" containerName="main"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586372    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbdc811-65cc-4aba-8be5-5a0150bd4402" containerName="main"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586507    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7dc019-30bf-4aa9-a2f4-91699d26dfdb" containerName="payload-pre-processing"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586517    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbdc811-65cc-4aba-8be5-5a0150bd4402" containerName="main"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586527    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d900bc8d-4020-4ec0-8dbe-1dc23712633b" containerName="payload-processing"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586553    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586559    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe31d77-e37f-41b1-87fb-8a3eb5374f40" containerName="istio-proxy"
Oct 30 21:04:58.586564 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.586566    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8812bf72-cfe9-4e13-9196-96e2ab52b2fd" containerName="payload-pre-processing"
Oct 30 21:04:58.591398 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.591381    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.632789 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.632756    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-home\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.632988 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.632798    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-dshm\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.633060 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.632981    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-tmp-dir\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.633060 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.633025    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da76b40-3caf-4cc2-99db-63cf46d612b7-tls-certs\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.633180 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.633130    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc298\" (UniqueName: \"kubernetes.io/projected/9da76b40-3caf-4cc2-99db-63cf46d612b7-kube-api-access-tc298\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.633180 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.633166    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-model-cache\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.648602 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.648566    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a064b4\"/\"kube-root-ca.crt\""
Oct 30 21:04:58.648602 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.648587    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a064b4\"/\"openshift-service-ca.crt\""
Oct 30 21:04:58.698852 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.698821    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a064b4\"/\"default-dockercfg-9f4nq\""
Oct 30 21:04:58.699419 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.699397    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a064b4\"/\"test-model-a064b4-kserve-self-signed-certs\""
Oct 30 21:04:58.702561 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.702524    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"]
Oct 30 21:04:58.718691 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.718666    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:58.734030 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.733990    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-tmp-dir\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734190 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734037    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da76b40-3caf-4cc2-99db-63cf46d612b7-tls-certs\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734190 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734105    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc298\" (UniqueName: \"kubernetes.io/projected/9da76b40-3caf-4cc2-99db-63cf46d612b7-kube-api-access-tc298\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734190 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734136    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-model-cache\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734388 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734261    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-home\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734388 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734296    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-dshm\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734392    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-tmp-dir\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734572 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734525    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-model-cache\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.734718 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.734702    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-home\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.736581 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.736554    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-dshm\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.736737 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.736719    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da76b40-3caf-4cc2-99db-63cf46d612b7-tls-certs\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.791811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.791777    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc298\" (UniqueName: \"kubernetes.io/projected/9da76b40-3caf-4cc2-99db-63cf46d612b7-kube-api-access-tc298\") pod \"test-model-a064b4-kserve-86c945f9b5-lgj4r\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") " pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.903811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.903720    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:04:58.904341 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:58.904318    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:59.240555 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:59.240507    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:59.295045 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:59.295016    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"]
Oct 30 21:04:59.296052 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:04:59.296022    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9da76b40_3caf_4cc2_99db_63cf46d612b7.slice/crio-f150e96978d6bd7ec8f58218363aa3e690cd8de59a550b4aa2ae607fe1e79435 WatchSource:0}: Error finding container f150e96978d6bd7ec8f58218363aa3e690cd8de59a550b4aa2ae607fe1e79435: Status 404 returned error can't find the container with id f150e96978d6bd7ec8f58218363aa3e690cd8de59a550b4aa2ae607fe1e79435
Oct 30 21:04:59.461393 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:59.461360    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:59.740356 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:59.740318    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r" event={"ID":"9da76b40-3caf-4cc2-99db-63cf46d612b7","Type":"ContainerStarted","Data":"5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb"}
Oct 30 21:04:59.740356 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:59.740360    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r" event={"ID":"9da76b40-3caf-4cc2-99db-63cf46d612b7","Type":"ContainerStarted","Data":"f150e96978d6bd7ec8f58218363aa3e690cd8de59a550b4aa2ae607fe1e79435"}
Oct 30 21:04:59.862152 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:59.862099    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r" podStartSLOduration=1.862075033 podStartE2EDuration="1.862075033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:59.859502904 +0000 UTC m=+2726.008553891" watchObservedRunningTime="2026-10-30 21:04:59.862075033 +0000 UTC m=+2726.011126021"
Oct 30 21:04:59.904242 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:04:59.904198    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:05:00.113097 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:00.113014    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:00.364224 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:00.364133    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:00.706611 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:00.706490    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:00.882830 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:00.882793    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:01.365714 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:01.365685    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:05.924393 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:05.924355    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:06.531158 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:06.531123    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:06.710461 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:06.710426    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:06.898167 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:06.897031    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:07.116267 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:05:07.116227    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:05:08.012735 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:08.012702    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:08.352352 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:08.352265    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:08.369675 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:08.369647    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"]
Oct 30 21:05:08.369883 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:08.369860    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy" containerID="cri-o://d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97" gracePeriod=30
Oct 30 21:05:08.543297 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:08.543264    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:08.904029 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:08.903996    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:05:08.917169 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:08.917146    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:05:09.701455 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:09.701426    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:09.821023 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:09.820994    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:05:09.874739 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:09.874709    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:10.883146 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:10.883112    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:11.260411 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:11.260373    5002 patch_prober.go:29] interesting pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:05:11.260621 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:11.260440    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:05:13.642318 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.642257    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 21:05:13.716398 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716371    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-token\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-token\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716427    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-data\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716463    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-envoy\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716561    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-certs\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716715 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716647    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-kube-api-access-88cvz\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-kube-api-access-88cvz\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716715 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716671    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-socket\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716715 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716710    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-credential-socket\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716876 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716742    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89051f8-4d46-4f57-9fb0-66535ddfd473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89051f8-4d46-4f57-9fb0-66535ddfd473-istiod-ca-cert\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716876 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716770    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-podinfo\") pod \"d89051f8-4d46-4f57-9fb0-66535ddfd473\" (UID: \"d89051f8-4d46-4f57-9fb0-66535ddfd473\") "
Oct 30 21:05:13.716876 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716790    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-data" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:13.717078 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.716877    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-certs" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:13.717199 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.717172    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-credential-socket" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:13.717267 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.717220    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89051f8-4d46-4f57-9fb0-66535ddfd473-istiod-ca-cert" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:13.717267 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.717238    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.717267 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.717254    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-socket" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:13.717787 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.717258    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.719101 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.719072    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-kube-api-access-88cvz" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "kube-api-access-88cvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:13.719101 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.719080    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-podinfo" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:05:13.719288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.719193    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-envoy" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:13.719288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.719266    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-token" pod "d89051f8-4d46-4f57-9fb0-66535ddfd473" (UID: "d89051f8-4d46-4f57-9fb0-66535ddfd473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:13.818441 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.818405    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88cvz\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-kube-api-access-88cvz\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.818441 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.818435    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.818683 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.818449    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.818683 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.818462    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89051f8-4d46-4f57-9fb0-66535ddfd473-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.818683 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.818474    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.818683 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.818485    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.818683 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.818497    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89051f8-4d46-4f57-9fb0-66535ddfd473-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.838326 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.838294    5002 generic.go:356] "Generic (PLEG): container finished" podID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerID="d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97" exitCode=0
Oct 30 21:05:13.838497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.838354    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"
Oct 30 21:05:13.838497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.838384    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" event={"ID":"d89051f8-4d46-4f57-9fb0-66535ddfd473","Type":"ContainerDied","Data":"d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97"}
Oct 30 21:05:13.838497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.838445    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv" event={"ID":"d89051f8-4d46-4f57-9fb0-66535ddfd473","Type":"ContainerDied","Data":"7c9115aad091872b887bd982b218dee3b86b5a955a4c4cd9bef2c5fed655e88f"}
Oct 30 21:05:13.838497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.838469    5002 scope.go:122] "RemoveContainer" containerID="d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97"
Oct 30 21:05:13.849496 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.849473    5002 scope.go:122] "RemoveContainer" containerID="204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702"
Oct 30 21:05:13.861328 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.861313    5002 scope.go:122] "RemoveContainer" containerID="d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97"
Oct 30 21:05:13.861631 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:05:13.861609    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97\": container with ID starting with d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97 not found: ID does not exist" containerID="d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97"
Oct 30 21:05:13.861702 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.861641    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97"} err="failed to get container status \"d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97\": rpc error: code = NotFound desc = could not find container \"d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97\": container with ID starting with d726d850dce31e5ff9e8e9e03fc940a8ff3364754a6d65a74bcaa30d64ee9b97 not found: ID does not exist"
Oct 30 21:05:13.861702 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.861660    5002 scope.go:122] "RemoveContainer" containerID="204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702"
Oct 30 21:05:13.861932 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:05:13.861914    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702\": container with ID starting with 204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702 not found: ID does not exist" containerID="204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702"
Oct 30 21:05:13.861995 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:13.861940    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702"} err="failed to get container status \"204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702\": rpc error: code = NotFound desc = could not find container \"204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702\": container with ID starting with 204f452ce64cb4f12b8b9e8d8220894c4b08fb0e5aaa70a2267e4c78f6928702 not found: ID does not exist"
Oct 30 21:05:14.006788 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:14.006637    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"]
Oct 30 21:05:14.010170 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:14.010147    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv"]
Oct 30 21:05:15.787352 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:15.787305    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:15.955215 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:15.955168    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:15.966388 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:15.966361    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" path="/var/lib/kubelet/pods/d89051f8-4d46-4f57-9fb0-66535ddfd473/volumes"
Oct 30 21:05:20.476414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:20.476382    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:20.618590 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:20.618550    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"]
Oct 30 21:05:20.728026 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:20.727937    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"]
Oct 30 21:05:21.022340 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:21.022260    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:21.162848 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:21.162818    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:05:21.196334 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:21.196297    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:25.203964 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.203931    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:25.618901 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.618850    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh" podUID="c784da18-e7c2-4288-b725-7ca1b42450e2" containerName="payload-processing" containerID="cri-o://cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a" gracePeriod=25
Oct 30 21:05:25.728492 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.728447    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" podUID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" containerName="payload-pre-processing" containerID="cri-o://85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf" gracePeriod=25
Oct 30 21:05:25.895573 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.895550    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 21:05:25.920493 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.920459    5002 generic.go:356] "Generic (PLEG): container finished" podID="c784da18-e7c2-4288-b725-7ca1b42450e2" containerID="cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a" exitCode=0
Oct 30 21:05:25.920698 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.920505    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"
Oct 30 21:05:25.920698 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.920573    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh" event={"ID":"c784da18-e7c2-4288-b725-7ca1b42450e2","Type":"ContainerDied","Data":"cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a"}
Oct 30 21:05:25.920698 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.920655    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh" event={"ID":"c784da18-e7c2-4288-b725-7ca1b42450e2","Type":"ContainerDied","Data":"f57ee6897c7fb02d3b5153cab98ba0a37a1fcf0fe3b2d8b65c206d4bf5526e8e"}
Oct 30 21:05:25.920698 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.920685    5002 scope.go:122] "RemoveContainer" containerID="cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a"
Oct 30 21:05:25.930982 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.930965    5002 scope.go:122] "RemoveContainer" containerID="cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a"
Oct 30 21:05:25.931241 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:05:25.931213    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a\": container with ID starting with cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a not found: ID does not exist" containerID="cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a"
Oct 30 21:05:25.931364 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:25.931245    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a"} err="failed to get container status \"cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a\": rpc error: code = NotFound desc = could not find container \"cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a\": container with ID starting with cf7e8e59f70ab4a25a4c6b5a21b85f1f5b0ad5343dfd32c75ff6bb089c09465a not found: ID does not exist"
Oct 30 21:05:26.056492 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.056457    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c784da18-e7c2-4288-b725-7ca1b42450e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784da18-e7c2-4288-b725-7ca1b42450e2-plugins-config-volume\") pod \"c784da18-e7c2-4288-b725-7ca1b42450e2\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") "
Oct 30 21:05:26.056693 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.056500    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c784da18-e7c2-4288-b725-7ca1b42450e2-kube-api-access-mpkbz\" (UniqueName: \"kubernetes.io/projected/c784da18-e7c2-4288-b725-7ca1b42450e2-kube-api-access-mpkbz\") pod \"c784da18-e7c2-4288-b725-7ca1b42450e2\" (UID: \"c784da18-e7c2-4288-b725-7ca1b42450e2\") "
Oct 30 21:05:26.057153 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.056845    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c784da18-e7c2-4288-b725-7ca1b42450e2-plugins-config-volume" pod "c784da18-e7c2-4288-b725-7ca1b42450e2" (UID: "c784da18-e7c2-4288-b725-7ca1b42450e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:26.057153 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.057081    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784da18-e7c2-4288-b725-7ca1b42450e2-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:26.058796 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.058775    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c784da18-e7c2-4288-b725-7ca1b42450e2-kube-api-access-mpkbz" pod "c784da18-e7c2-4288-b725-7ca1b42450e2" (UID: "c784da18-e7c2-4288-b725-7ca1b42450e2"). InnerVolumeSpecName "kube-api-access-mpkbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:26.157789 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.157689    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpkbz\" (UniqueName: \"kubernetes.io/projected/c784da18-e7c2-4288-b725-7ca1b42450e2-kube-api-access-mpkbz\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:26.342185 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.342149    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"]
Oct 30 21:05:26.347762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:26.347735    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh"]
Oct 30 21:05:27.967098 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:27.967054    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c784da18-e7c2-4288-b725-7ca1b42450e2" path="/var/lib/kubelet/pods/c784da18-e7c2-4288-b725-7ca1b42450e2/volumes"
Oct 30 21:05:28.739437 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:28.739395    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" podUID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:05:29.880494 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:29.880463    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:33.845195 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:33.845154    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"]
Oct 30 21:05:33.845747 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:33.845417    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq" podUID="e430c6d2-88ee-413f-8b7d-f90efe62a92f" containerName="maas-api" containerID="cri-o://4cffe95c08b985318b7cc8865efb0296882a70badfd96ace8cb6e36fd9dfbed5" gracePeriod=30
Oct 30 21:05:33.980514 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:33.980482    5002 generic.go:356] "Generic (PLEG): container finished" podID="e430c6d2-88ee-413f-8b7d-f90efe62a92f" containerID="4cffe95c08b985318b7cc8865efb0296882a70badfd96ace8cb6e36fd9dfbed5" exitCode=0
Oct 30 21:05:33.980817 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:33.980795    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq" event={"ID":"e430c6d2-88ee-413f-8b7d-f90efe62a92f","Type":"ContainerDied","Data":"4cffe95c08b985318b7cc8865efb0296882a70badfd96ace8cb6e36fd9dfbed5"}
Oct 30 21:05:33.983163 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:33.983141    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:34.120825 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.120796    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 21:05:34.247357 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.247321    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e430c6d2-88ee-413f-8b7d-f90efe62a92f-kube-api-access-2ghvv\" (UniqueName: \"kubernetes.io/projected/e430c6d2-88ee-413f-8b7d-f90efe62a92f-kube-api-access-2ghvv\") pod \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") "
Oct 30 21:05:34.247598 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.247407    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls\") pod \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") "
Oct 30 21:05:34.247598 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.247473    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e430c6d2-88ee-413f-8b7d-f90efe62a92f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430c6d2-88ee-413f-8b7d-f90efe62a92f-openshift-service-ca\") pod \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") "
Oct 30 21:05:34.247598 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.247516    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-metrics-certs\") pod \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\" (UID: \"e430c6d2-88ee-413f-8b7d-f90efe62a92f\") "
Oct 30 21:05:34.247959 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.247930    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e430c6d2-88ee-413f-8b7d-f90efe62a92f-openshift-service-ca" pod "e430c6d2-88ee-413f-8b7d-f90efe62a92f" (UID: "e430c6d2-88ee-413f-8b7d-f90efe62a92f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:34.248102 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.248080    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430c6d2-88ee-413f-8b7d-f90efe62a92f-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:34.250375 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.250353    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls" pod "e430c6d2-88ee-413f-8b7d-f90efe62a92f" (UID: "e430c6d2-88ee-413f-8b7d-f90efe62a92f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:34.251064 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.251030    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e430c6d2-88ee-413f-8b7d-f90efe62a92f-kube-api-access-2ghvv" pod "e430c6d2-88ee-413f-8b7d-f90efe62a92f" (UID: "e430c6d2-88ee-413f-8b7d-f90efe62a92f"). InnerVolumeSpecName "kube-api-access-2ghvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:34.258736 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.258710    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-metrics-certs" pod "e430c6d2-88ee-413f-8b7d-f90efe62a92f" (UID: "e430c6d2-88ee-413f-8b7d-f90efe62a92f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:34.349970 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.349937    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:34.349970 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.349977    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghvv\" (UniqueName: \"kubernetes.io/projected/e430c6d2-88ee-413f-8b7d-f90efe62a92f-kube-api-access-2ghvv\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:34.350180 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.349992    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e430c6d2-88ee-413f-8b7d-f90efe62a92f-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:34.989605 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.989565    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq" event={"ID":"e430c6d2-88ee-413f-8b7d-f90efe62a92f","Type":"ContainerDied","Data":"63a4504b7e4ad609c808fef8ef167f95cc35ca95a83fcefa26df1d02b5e23ff9"}
Oct 30 21:05:34.990052 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.989632    5002 scope.go:122] "RemoveContainer" containerID="4cffe95c08b985318b7cc8865efb0296882a70badfd96ace8cb6e36fd9dfbed5"
Oct 30 21:05:34.990052 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:34.989581    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"
Oct 30 21:05:35.157856 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:35.157813    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"]
Oct 30 21:05:35.206589 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:35.206556    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11a0b0-c6bc6749d-4ckdq"]
Oct 30 21:05:35.504836 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:35.504805    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:35.776069 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:35.775979    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:35.963644 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:35.963613    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:35.966315 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:35.966292    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e430c6d2-88ee-413f-8b7d-f90efe62a92f" path="/var/lib/kubelet/pods/e430c6d2-88ee-413f-8b7d-f90efe62a92f/volumes"
Oct 30 21:05:36.421017 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:36.420986    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:37.115131 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:05:37.115102    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:05:38.739320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:38.739278    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" podUID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:05:39.691970 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:39.691935    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:40.885377 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:40.885341    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:42.608150 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:42.608116    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:42.803511 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:42.803477    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:42.944288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:42.944209    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"]
Oct 30 21:05:42.944438 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:42.944408    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy" containerID="cri-o://6697fcc4065760059cc61045a3e7382a777aecebd75861f2e324918656b63b2d" gracePeriod=30
Oct 30 21:05:42.982492 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:42.982455    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:43.433527 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:43.433498    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:45.990361 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:45.990300    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 21:05:46.058771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.058736    5002 generic.go:356] "Generic (PLEG): container finished" podID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" containerID="85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf" exitCode=0
Oct 30 21:05:46.058952 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.058779    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" event={"ID":"c9725de7-41ec-4ebd-bde8-f9fe5c741209","Type":"ContainerDied","Data":"85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf"}
Oct 30 21:05:46.058952 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.058787    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"
Oct 30 21:05:46.058952 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.058826    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q" event={"ID":"c9725de7-41ec-4ebd-bde8-f9fe5c741209","Type":"ContainerDied","Data":"aae81923f71e9721149a0ea0ee89736e6226f497709eb5b2d8b034d7fa43e3bc"}
Oct 30 21:05:46.058952 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.058863    5002 scope.go:122] "RemoveContainer" containerID="85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf"
Oct 30 21:05:46.070402 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.070381    5002 scope.go:122] "RemoveContainer" containerID="85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf"
Oct 30 21:05:46.070711 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:05:46.070684    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf\": container with ID starting with 85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf not found: ID does not exist" containerID="85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf"
Oct 30 21:05:46.070814 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.070715    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf"} err="failed to get container status \"85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf\": rpc error: code = NotFound desc = could not find container \"85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf\": container with ID starting with 85618ab23573e254071feea1e8a919b252e7e928e741ed793aa0f0c0470c1fcf not found: ID does not exist"
Oct 30 21:05:46.072026 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.072006    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9725de7-41ec-4ebd-bde8-f9fe5c741209-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9725de7-41ec-4ebd-bde8-f9fe5c741209-plugins-config-volume\") pod \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") "
Oct 30 21:05:46.072186 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.072172    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9725de7-41ec-4ebd-bde8-f9fe5c741209-kube-api-access-rrxgh\" (UniqueName: \"kubernetes.io/projected/c9725de7-41ec-4ebd-bde8-f9fe5c741209-kube-api-access-rrxgh\") pod \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\" (UID: \"c9725de7-41ec-4ebd-bde8-f9fe5c741209\") "
Oct 30 21:05:46.072395 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.072373    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9725de7-41ec-4ebd-bde8-f9fe5c741209-plugins-config-volume" pod "c9725de7-41ec-4ebd-bde8-f9fe5c741209" (UID: "c9725de7-41ec-4ebd-bde8-f9fe5c741209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:46.072516 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.072501    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9725de7-41ec-4ebd-bde8-f9fe5c741209-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:46.074516 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.074497    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9725de7-41ec-4ebd-bde8-f9fe5c741209-kube-api-access-rrxgh" pod "c9725de7-41ec-4ebd-bde8-f9fe5c741209" (UID: "c9725de7-41ec-4ebd-bde8-f9fe5c741209"). InnerVolumeSpecName "kube-api-access-rrxgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:46.123496 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.123462    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:46.173252 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.173217    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrxgh\" (UniqueName: \"kubernetes.io/projected/c9725de7-41ec-4ebd-bde8-f9fe5c741209-kube-api-access-rrxgh\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:46.482676 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.482646    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"]
Oct 30 21:05:46.532909 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:46.532867    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q"]
Oct 30 21:05:47.603356 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:47.603323    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:05:47.969104 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:47.969077    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" path="/var/lib/kubelet/pods/c9725de7-41ec-4ebd-bde8-f9fe5c741209/volumes"
Oct 30 21:05:48.077839 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.077794    5002 generic.go:356] "Generic (PLEG): container finished" podID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerID="6697fcc4065760059cc61045a3e7382a777aecebd75861f2e324918656b63b2d" exitCode=0
Oct 30 21:05:48.078003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.077863    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" event={"ID":"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f","Type":"ContainerDied","Data":"6697fcc4065760059cc61045a3e7382a777aecebd75861f2e324918656b63b2d"}
Oct 30 21:05:48.078003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.077985    5002 scope.go:122] "RemoveContainer" containerID="e45614f34112a5c67e2aec0fb098cc08e6ea342cbb92aa2eceaf9eedc2e9cce3"
Oct 30 21:05:48.219993 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.219925    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 21:05:48.295573 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295514    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-podinfo\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.295762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295592    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-data\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.295762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295628    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-credential-socket\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.295762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295649    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-envoy\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.295762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295705    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-kube-api-access-gjssf\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-kube-api-access-gjssf\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.296041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295835    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-certs\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.296041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295887    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-data" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:48.296041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295893    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-token\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-token\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.296041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295934    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-credential-socket" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:48.296041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295962    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istiod-ca-cert\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.296041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.295985    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-socket\") pod \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\" (UID: \"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f\") "
Oct 30 21:05:48.296677 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.296510    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istiod-ca-cert" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:48.296677 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.296638    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.296677 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.296659    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.296677 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.296656    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-socket" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:48.297028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.296862    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-certs" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:48.298158 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.298134    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-envoy" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:48.298406 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.298381    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-podinfo" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:05:48.298663 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.298639    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-token" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:48.298753 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.298665    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-kube-api-access-gjssf" pod "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" (UID: "00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f"). InnerVolumeSpecName "kube-api-access-gjssf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:48.398183 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.398141    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjssf\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-kube-api-access-gjssf\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.398183 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.398172    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.398183 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.398182    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.398183 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.398190    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.398686 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.398200    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.398686 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.398209    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.398686 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.398217    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:48.482351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:48.482266    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:49.086331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:49.086305    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"
Oct 30 21:05:49.086331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:49.086313    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6" event={"ID":"00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f","Type":"ContainerDied","Data":"d7daa2d7e86b86aa907e74906d425ba29749405cbee24b83f1b30e9db2067591"}
Oct 30 21:05:49.086865 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:49.086364    5002 scope.go:122] "RemoveContainer" containerID="6697fcc4065760059cc61045a3e7382a777aecebd75861f2e324918656b63b2d"
Oct 30 21:05:49.272192 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:49.272158    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"]
Oct 30 21:05:49.322517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:49.322487    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6"]
Oct 30 21:05:49.967514 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:49.967479    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" path="/var/lib/kubelet/pods/00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f/volumes"
Oct 30 21:05:50.979428 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:50.979393    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:55.712120 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:55.712085    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:55.891738 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:55.891631    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:56.063776 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:05:56.063694    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:00.408365 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:00.408313    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:00.608664 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:00.608633    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:00.793905 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:00.793866    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:05.088786 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:05.088738    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:07.116410 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:06:07.116368    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:06:07.543418 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:07.543376    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"]
Oct 30 21:06:07.543701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:07.543653    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy" containerID="cri-o://43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9" gracePeriod=30
Oct 30 21:06:09.989417 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:09.989374    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:12.800726 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.800700    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 21:06:12.854886 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.854817    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-data\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.854886 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.854856    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-socket\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.854886 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.854889    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-kube-api-access-w7btm\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-kube-api-access-w7btm\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.854926    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-certs\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.854955    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-credential-socket\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.854977    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istiod-ca-cert\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855008    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-podinfo\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855115    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-token\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-token\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855129    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-socket" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855140    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-data" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:12.855159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855156    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-envoy\") pod \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\" (UID: \"a1286f6f-71b5-499b-80ae-aafcf82c9c8a\") "
Oct 30 21:06:12.855597 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855287    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-certs" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:12.855597 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855460    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istiod-ca-cert" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:12.855699 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855609    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-credential-socket" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:12.855699 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855629    5002 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.855699 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855652    5002 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istiod-ca-cert\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.855699 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855665    5002 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-data\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.855699 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.855680    5002 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-workload-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.857661 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.857623    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-kube-api-access-w7btm" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "kube-api-access-w7btm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:12.857799 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.857733    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-envoy" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:12.857930 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.857909    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-token" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:12.858015 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.857953    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-podinfo" pod "a1286f6f-71b5-499b-80ae-aafcf82c9c8a" (UID: "a1286f6f-71b5-499b-80ae-aafcf82c9c8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:12.956231 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.956200    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7btm\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-kube-api-access-w7btm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.956231 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.956225    5002 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-credential-socket\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.956231 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.956236    5002 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-podinfo\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.956467 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.956246    5002 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-token\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.956467 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:12.956254    5002 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1286f6f-71b5-499b-80ae-aafcf82c9c8a-istio-envoy\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.237333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.237292    5002 generic.go:356] "Generic (PLEG): container finished" podID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerID="43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9" exitCode=0
Oct 30 21:06:13.237513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.237347    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" event={"ID":"a1286f6f-71b5-499b-80ae-aafcf82c9c8a","Type":"ContainerDied","Data":"43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9"}
Oct 30 21:06:13.237513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.237432    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m" event={"ID":"a1286f6f-71b5-499b-80ae-aafcf82c9c8a","Type":"ContainerDied","Data":"2bbd4401873dd21a18a090a355c2bc26249563f79f53fe0b33ffa46aa831c629"}
Oct 30 21:06:13.237513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.237369    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"
Oct 30 21:06:13.237513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.237465    5002 scope.go:122] "RemoveContainer" containerID="43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9"
Oct 30 21:06:13.249867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.249474    5002 scope.go:122] "RemoveContainer" containerID="43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9"
Oct 30 21:06:13.249867 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:06:13.249842    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9\": container with ID starting with 43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9 not found: ID does not exist" containerID="43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9"
Oct 30 21:06:13.249867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.249881    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9"} err="failed to get container status \"43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9\": rpc error: code = NotFound desc = could not find container \"43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9\": container with ID starting with 43c4e3c79667c843fad978969c78f0fcac149270b197eb8363d98876c9902be9 not found: ID does not exist"
Oct 30 21:06:13.405251 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.405215    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"]
Oct 30 21:06:13.455769 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.455732    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m"]
Oct 30 21:06:13.966336 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:13.966305    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" path="/var/lib/kubelet/pods/a1286f6f-71b5-499b-80ae-aafcf82c9c8a/volumes"
Oct 30 21:06:14.685793 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:14.685747    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:15.460415 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:15.460372    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:06:19.476962 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:19.476901    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:24.187485 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:24.187453    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:25.211505 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:25.211452    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:25.385794 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:25.385759    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:25.940389 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:25.940350    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:28.880310 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:28.880276    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:31.685915 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:31.685864    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:31.869587 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:31.869547    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:32.820468 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:32.820433    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"]
Oct 30 21:06:32.820987 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:32.820654    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc" podUID="991c5524-1638-4f50-ab27-289d7ab6824a" containerName="maas-api" containerID="cri-o://887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f" gracePeriod=30
Oct 30 21:06:32.963134 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:32.963072    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:33.087235 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.087172    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 21:06:33.196835 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.190676    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991c5524-1638-4f50-ab27-289d7ab6824a-kube-api-access-vzf4t\" (UniqueName: \"kubernetes.io/projected/991c5524-1638-4f50-ab27-289d7ab6824a-kube-api-access-vzf4t\") pod \"991c5524-1638-4f50-ab27-289d7ab6824a\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") "
Oct 30 21:06:33.196835 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.190747    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-metrics-certs\") pod \"991c5524-1638-4f50-ab27-289d7ab6824a\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") "
Oct 30 21:06:33.196835 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.190830    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls\") pod \"991c5524-1638-4f50-ab27-289d7ab6824a\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") "
Oct 30 21:06:33.196835 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.190952    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991c5524-1638-4f50-ab27-289d7ab6824a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/991c5524-1638-4f50-ab27-289d7ab6824a-openshift-service-ca\") pod \"991c5524-1638-4f50-ab27-289d7ab6824a\" (UID: \"991c5524-1638-4f50-ab27-289d7ab6824a\") "
Oct 30 21:06:33.196835 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.193896    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991c5524-1638-4f50-ab27-289d7ab6824a-openshift-service-ca" pod "991c5524-1638-4f50-ab27-289d7ab6824a" (UID: "991c5524-1638-4f50-ab27-289d7ab6824a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:33.198799 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.198753    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-metrics-certs" pod "991c5524-1638-4f50-ab27-289d7ab6824a" (UID: "991c5524-1638-4f50-ab27-289d7ab6824a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:33.199025 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.199000    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls" pod "991c5524-1638-4f50-ab27-289d7ab6824a" (UID: "991c5524-1638-4f50-ab27-289d7ab6824a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:33.199243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.199215    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991c5524-1638-4f50-ab27-289d7ab6824a-kube-api-access-vzf4t" pod "991c5524-1638-4f50-ab27-289d7ab6824a" (UID: "991c5524-1638-4f50-ab27-289d7ab6824a"). InnerVolumeSpecName "kube-api-access-vzf4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:33.292700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.292661    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/991c5524-1638-4f50-ab27-289d7ab6824a-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:33.292700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.292696    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzf4t\" (UniqueName: \"kubernetes.io/projected/991c5524-1638-4f50-ab27-289d7ab6824a-kube-api-access-vzf4t\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:33.292700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.292708    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:33.292956 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.292720    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/991c5524-1638-4f50-ab27-289d7ab6824a-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:33.357647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.357521    5002 generic.go:356] "Generic (PLEG): container finished" podID="991c5524-1638-4f50-ab27-289d7ab6824a" containerID="887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f" exitCode=0
Oct 30 21:06:33.357647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.357594    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"
Oct 30 21:06:33.357647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.357613    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc" event={"ID":"991c5524-1638-4f50-ab27-289d7ab6824a","Type":"ContainerDied","Data":"887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f"}
Oct 30 21:06:33.357935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.357668    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc" event={"ID":"991c5524-1638-4f50-ab27-289d7ab6824a","Type":"ContainerDied","Data":"cfeee41886717f3d40a88614e841d09a7aed1360f0ce3b6dffa4443904b713d1"}
Oct 30 21:06:33.357935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.357698    5002 scope.go:122] "RemoveContainer" containerID="887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f"
Oct 30 21:06:33.367907 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.367885    5002 scope.go:122] "RemoveContainer" containerID="887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f"
Oct 30 21:06:33.368177 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:06:33.368153    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f\": container with ID starting with 887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f not found: ID does not exist" containerID="887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f"
Oct 30 21:06:33.368246 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.368186    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f"} err="failed to get container status \"887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f\": rpc error: code = NotFound desc = could not find container \"887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f\": container with ID starting with 887a6e8c49cab78689f43c2fc69532d3e241c6b2a536073bd93ac5473fb48c0f not found: ID does not exist"
Oct 30 21:06:33.522779 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.522743    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"]
Oct 30 21:06:33.525065 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.525039    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0a78ac-665f949785-fs2qc"]
Oct 30 21:06:33.965866 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:33.965832    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991c5524-1638-4f50-ab27-289d7ab6824a" path="/var/lib/kubelet/pods/991c5524-1638-4f50-ab27-289d7ab6824a/volumes"
Oct 30 21:06:34.466445 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:34.466411    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:34.695874 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:34.695834    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:34.855752 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:34.855677    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:06:34.869217 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:34.869178    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:35.051274 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:35.051240    5002 scope.go:122] "RemoveContainer" containerID="f8ff63f641593021d28aab3c34000ec2a8e2425f1fe0a87afd62f8168d0e2ffc"
Oct 30 21:06:35.380334 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:35.380299    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:37.115025 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:06:37.114988    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:06:39.700954 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:39.700898    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:40.036028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:40.035992    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:41.334930 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:41.334889    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:41.510717 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:41.510677    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:41.708117 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:41.708013    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:44.393726 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:44.393638    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:49.088421 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:49.088384    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:53.791255 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:53.791222    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:54.910346 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:54.910311    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:06:58.598909 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:06:58.598867    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:03.278144 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:03.278103    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:07.116042 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:07:07.116002    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:07:07.992374 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:07.992339    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:08.433237 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:08.433146    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:07:12.688098 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:12.688061    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:17.392826 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:17.392786    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:22.103148 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:22.103092    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:26.784923 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:26.784882    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:30.670969 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:30.670935    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:31.586120 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:31.586065    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:36.291557 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:36.291508    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:37.115520 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:07:37.115481    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:07:39.879657 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:39.879612    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:40.377206 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:40.377170    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:40.978320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:40.978288    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:44.651036 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:44.650996    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:07:45.705973 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:45.705939    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:48.313218 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:48.313181    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:48.502300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:48.502271    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:50.387342 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:50.387312    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:55.084086 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:55.084051    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:59.782696 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:07:59.782633    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:02.735779 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:02.735742    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:08:04.489133 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:04.489084    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:07.116511 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:08:07.116471    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:08:08.063888 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:08.063849    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"]
Oct 30 21:08:08.123455 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:08.123416    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:09.241718 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:09.241683    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:14.507953 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:14.507841    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:14.934338 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:14.934254    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:08:19.188861 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:19.188820    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:20.516855 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:20.516815    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:20.699783 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:20.699741    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:23.087275 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.087231    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r" podUID="9da76b40-3caf-4cc2-99db-63cf46d612b7" containerName="main" containerID="cri-o://5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb" gracePeriod=45
Oct 30 21:08:23.357366 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.357333    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:08:23.452852 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.452810    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da76b40-3caf-4cc2-99db-63cf46d612b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/9da76b40-3caf-4cc2-99db-63cf46d612b7-tls-certs\") pod \"9da76b40-3caf-4cc2-99db-63cf46d612b7\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") "
Oct 30 21:08:23.453053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.452873    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-model-cache\") pod \"9da76b40-3caf-4cc2-99db-63cf46d612b7\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") "
Oct 30 21:08:23.453053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.452899    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-home\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-home\") pod \"9da76b40-3caf-4cc2-99db-63cf46d612b7\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") "
Oct 30 21:08:23.453053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.452974    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da76b40-3caf-4cc2-99db-63cf46d612b7-kube-api-access-tc298\" (UniqueName: \"kubernetes.io/projected/9da76b40-3caf-4cc2-99db-63cf46d612b7-kube-api-access-tc298\") pod \"9da76b40-3caf-4cc2-99db-63cf46d612b7\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") "
Oct 30 21:08:23.453053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.453028    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-dshm\") pod \"9da76b40-3caf-4cc2-99db-63cf46d612b7\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") "
Oct 30 21:08:23.453306 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.453117    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-tmp-dir\") pod \"9da76b40-3caf-4cc2-99db-63cf46d612b7\" (UID: \"9da76b40-3caf-4cc2-99db-63cf46d612b7\") "
Oct 30 21:08:23.453306 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.453183    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-model-cache" pod "9da76b40-3caf-4cc2-99db-63cf46d612b7" (UID: "9da76b40-3caf-4cc2-99db-63cf46d612b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:23.453306 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.453219    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-home" pod "9da76b40-3caf-4cc2-99db-63cf46d612b7" (UID: "9da76b40-3caf-4cc2-99db-63cf46d612b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:23.453508 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.453471    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-tmp-dir" pod "9da76b40-3caf-4cc2-99db-63cf46d612b7" (UID: "9da76b40-3caf-4cc2-99db-63cf46d612b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:23.453657 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.453559    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:23.453657 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.453582    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:23.455682 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.455653    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da76b40-3caf-4cc2-99db-63cf46d612b7-kube-api-access-tc298" pod "9da76b40-3caf-4cc2-99db-63cf46d612b7" (UID: "9da76b40-3caf-4cc2-99db-63cf46d612b7"). InnerVolumeSpecName "kube-api-access-tc298". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:23.455782 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.455689    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da76b40-3caf-4cc2-99db-63cf46d612b7-tls-certs" pod "9da76b40-3caf-4cc2-99db-63cf46d612b7" (UID: "9da76b40-3caf-4cc2-99db-63cf46d612b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:23.455782 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.455707    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-dshm" pod "9da76b40-3caf-4cc2-99db-63cf46d612b7" (UID: "9da76b40-3caf-4cc2-99db-63cf46d612b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:23.554376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.554333    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc298\" (UniqueName: \"kubernetes.io/projected/9da76b40-3caf-4cc2-99db-63cf46d612b7-kube-api-access-tc298\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:23.554376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.554367    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:23.554376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.554377    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da76b40-3caf-4cc2-99db-63cf46d612b7-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:23.554376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.554385    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da76b40-3caf-4cc2-99db-63cf46d612b7-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:23.793971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:23.793917    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:24.004107 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.004063    5002 generic.go:356] "Generic (PLEG): container finished" podID="9da76b40-3caf-4cc2-99db-63cf46d612b7" containerID="5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb" exitCode=0
Oct 30 21:08:24.004320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.004122    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"
Oct 30 21:08:24.004320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.004143    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r" event={"ID":"9da76b40-3caf-4cc2-99db-63cf46d612b7","Type":"ContainerDied","Data":"5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb"}
Oct 30 21:08:24.004320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.004189    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r" event={"ID":"9da76b40-3caf-4cc2-99db-63cf46d612b7","Type":"ContainerDied","Data":"f150e96978d6bd7ec8f58218363aa3e690cd8de59a550b4aa2ae607fe1e79435"}
Oct 30 21:08:24.004320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.004208    5002 scope.go:122] "RemoveContainer" containerID="5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb"
Oct 30 21:08:24.016109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.016086    5002 scope.go:122] "RemoveContainer" containerID="5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb"
Oct 30 21:08:24.016496 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:08:24.016471    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb\": container with ID starting with 5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb not found: ID does not exist" containerID="5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb"
Oct 30 21:08:24.016606 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.016503    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb"} err="failed to get container status \"5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb\": rpc error: code = NotFound desc = could not find container \"5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb\": container with ID starting with 5dce7226cc383a6ccc0ef250446558efc6dc510d2d87775cb0acef449b75f7eb not found: ID does not exist"
Oct 30 21:08:24.170668 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.170525    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"]
Oct 30 21:08:24.173400 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:24.173374    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a064b4/test-model-a064b4-kserve-86c945f9b5-lgj4r"]
Oct 30 21:08:25.967104 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:25.967067    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da76b40-3caf-4cc2-99db-63cf46d612b7" path="/var/lib/kubelet/pods/9da76b40-3caf-4cc2-99db-63cf46d612b7/volumes"
Oct 30 21:08:28.453518 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:28.453463    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:33.182719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:33.182684    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:36.434067 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:36.434028    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:37.117423 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:08:37.117381    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:08:37.140142 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.140105    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"]
Oct 30 21:08:37.140342 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.140304    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8" podUID="114a89a2-44c3-4622-90a3-827fd3651162" containerName="maas-api" containerID="cri-o://d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125" gracePeriod=30
Oct 30 21:08:37.271921 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.271881    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:37.433567 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.433498    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:08:37.497033 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.496992    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls\") pod \"114a89a2-44c3-4622-90a3-827fd3651162\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") "
Oct 30 21:08:37.497610 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.497068    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114a89a2-44c3-4622-90a3-827fd3651162-kube-api-access-prw79\" (UniqueName: \"kubernetes.io/projected/114a89a2-44c3-4622-90a3-827fd3651162-kube-api-access-prw79\") pod \"114a89a2-44c3-4622-90a3-827fd3651162\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") "
Oct 30 21:08:37.497610 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.497170    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114a89a2-44c3-4622-90a3-827fd3651162-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114a89a2-44c3-4622-90a3-827fd3651162-openshift-service-ca\") pod \"114a89a2-44c3-4622-90a3-827fd3651162\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") "
Oct 30 21:08:37.497610 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.497231    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-metrics-certs\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-metrics-certs\") pod \"114a89a2-44c3-4622-90a3-827fd3651162\" (UID: \"114a89a2-44c3-4622-90a3-827fd3651162\") "
Oct 30 21:08:37.497794 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.497668    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114a89a2-44c3-4622-90a3-827fd3651162-openshift-service-ca" pod "114a89a2-44c3-4622-90a3-827fd3651162" (UID: "114a89a2-44c3-4622-90a3-827fd3651162"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:37.499687 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.499650    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-metrics-certs" pod "114a89a2-44c3-4622-90a3-827fd3651162" (UID: "114a89a2-44c3-4622-90a3-827fd3651162"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:37.500070 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.500043    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls" pod "114a89a2-44c3-4622-90a3-827fd3651162" (UID: "114a89a2-44c3-4622-90a3-827fd3651162"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:37.500174 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.500066    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114a89a2-44c3-4622-90a3-827fd3651162-kube-api-access-prw79" pod "114a89a2-44c3-4622-90a3-827fd3651162" (UID: "114a89a2-44c3-4622-90a3-827fd3651162"). InnerVolumeSpecName "kube-api-access-prw79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:37.598206 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.598168    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114a89a2-44c3-4622-90a3-827fd3651162-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:37.598206 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.598209    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:37.598483 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.598219    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114a89a2-44c3-4622-90a3-827fd3651162-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:37.598483 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:37.598228    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prw79\" (UniqueName: \"kubernetes.io/projected/114a89a2-44c3-4622-90a3-827fd3651162-kube-api-access-prw79\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:38.091449 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.091411    5002 generic.go:356] "Generic (PLEG): container finished" podID="114a89a2-44c3-4622-90a3-827fd3651162" containerID="d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125" exitCode=0
Oct 30 21:08:38.091710 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.091481    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8" event={"ID":"114a89a2-44c3-4622-90a3-827fd3651162","Type":"ContainerDied","Data":"d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125"}
Oct 30 21:08:38.091710 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.091515    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"
Oct 30 21:08:38.091710 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.091565    5002 scope.go:122] "RemoveContainer" containerID="d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125"
Oct 30 21:08:38.091710 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.091520    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8" event={"ID":"114a89a2-44c3-4622-90a3-827fd3651162","Type":"ContainerDied","Data":"5ab889ca8c7c469b72062d91c6e6f8bb2db96268acc992338f14d434dc2b7460"}
Oct 30 21:08:38.103805 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.103757    5002 scope.go:122] "RemoveContainer" containerID="d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125"
Oct 30 21:08:38.104235 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:08:38.104213    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125\": container with ID starting with d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125 not found: ID does not exist" containerID="d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125"
Oct 30 21:08:38.104302 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.104244    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125"} err="failed to get container status \"d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125\": rpc error: code = NotFound desc = could not find container \"d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125\": container with ID starting with d82dfbb21814bdaea909d1d73fb7836a3cc1108a21e3d981a393e6faa7773125 not found: ID does not exist"
Oct 30 21:08:38.261293 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.261201    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"]
Oct 30 21:08:38.265073 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.265039    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2fa881-57884d8578-8w2n8"]
Oct 30 21:08:38.766808 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.766769    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:38.991692 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:38.991628    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:39.167486 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:39.167375    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:39.806724 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:39.806690    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:39.966271 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:39.966235    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114a89a2-44c3-4622-90a3-827fd3651162" path="/var/lib/kubelet/pods/114a89a2-44c3-4622-90a3-827fd3651162/volumes"
Oct 30 21:08:40.721383 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:40.721331    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:42.743019 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:42.742974    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:08:44.096508 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:44.096461    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:45.557500 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:45.557466    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:45.733124 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:45.733089    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:45.915888 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:45.915792    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:49.047191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:49.047155    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:53.907741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:53.907703    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:58.340493 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:58.340456    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"]
Oct 30 21:08:58.447335 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:58.447293    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"]
Oct 30 21:08:58.845549 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:58.845498    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:59.013410 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:08:59.013372    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:03.341729 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:03.341663    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" podUID="d15d64ca-0058-43ef-90a2-b879e816a641" containerName="payload-processing" containerID="cri-o://54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7" gracePeriod=25
Oct 30 21:09:03.424010 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:03.423973    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:03.447859 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:03.447822    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" podUID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" containerName="payload-pre-processing" containerID="cri-o://e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d" gracePeriod=25
Oct 30 21:09:04.926107 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:04.926056    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" podUID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:09:04.926107 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:04.926103    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" podUID="d15d64ca-0058-43ef-90a2-b879e816a641" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:09:07.117074 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:09:07.117033    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:09:07.421016 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.420922    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"]
Oct 30 21:09:07.421213 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.421136    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk" podUID="c98bdadb-0093-49eb-b21c-92b0c19fb4ac" containerName="maas-api" containerID="cri-o://8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f" gracePeriod=30
Oct 30 21:09:07.571248 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.571213    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:07.690890 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.690864    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:09:07.712191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.712158    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls\") pod \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") "
Oct 30 21:09:07.714546 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.714498    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls" pod "c98bdadb-0093-49eb-b21c-92b0c19fb4ac" (UID: "c98bdadb-0093-49eb-b21c-92b0c19fb4ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:07.815044 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.813908    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-openshift-service-ca\") pod \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") "
Oct 30 21:09:07.815044 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.813992    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-kube-api-access-g8945\" (UniqueName: \"kubernetes.io/projected/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-kube-api-access-g8945\") pod \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") "
Oct 30 21:09:07.815044 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.814071    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-metrics-certs\") pod \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\" (UID: \"c98bdadb-0093-49eb-b21c-92b0c19fb4ac\") "
Oct 30 21:09:07.815044 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.814470    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:07.815442 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.815278    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-openshift-service-ca" pod "c98bdadb-0093-49eb-b21c-92b0c19fb4ac" (UID: "c98bdadb-0093-49eb-b21c-92b0c19fb4ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:07.817859 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.817814    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-kube-api-access-g8945" pod "c98bdadb-0093-49eb-b21c-92b0c19fb4ac" (UID: "c98bdadb-0093-49eb-b21c-92b0c19fb4ac"). InnerVolumeSpecName "kube-api-access-g8945". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:07.821644 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.821601    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-metrics-certs" pod "c98bdadb-0093-49eb-b21c-92b0c19fb4ac" (UID: "c98bdadb-0093-49eb-b21c-92b0c19fb4ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:07.915636 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.915598    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:07.915636 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.915626    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:07.915912 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:07.915656    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8945\" (UniqueName: \"kubernetes.io/projected/c98bdadb-0093-49eb-b21c-92b0c19fb4ac-kube-api-access-g8945\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:08.277973 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.277867    5002 generic.go:356] "Generic (PLEG): container finished" podID="c98bdadb-0093-49eb-b21c-92b0c19fb4ac" containerID="8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f" exitCode=0
Oct 30 21:09:08.277973 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.277922    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"
Oct 30 21:09:08.277973 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.277959    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk" event={"ID":"c98bdadb-0093-49eb-b21c-92b0c19fb4ac","Type":"ContainerDied","Data":"8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f"}
Oct 30 21:09:08.278627 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.278012    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk" event={"ID":"c98bdadb-0093-49eb-b21c-92b0c19fb4ac","Type":"ContainerDied","Data":"66465fc4578fe2ef4d03e48c715d571e5827b3ccdc0aa6ee47558c8165fdb559"}
Oct 30 21:09:08.278627 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.278036    5002 scope.go:122] "RemoveContainer" containerID="8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f"
Oct 30 21:09:08.287776 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.287756    5002 scope.go:122] "RemoveContainer" containerID="8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f"
Oct 30 21:09:08.288055 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:09:08.288036    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f\": container with ID starting with 8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f not found: ID does not exist" containerID="8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f"
Oct 30 21:09:08.288105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.288067    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f"} err="failed to get container status \"8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f\": rpc error: code = NotFound desc = could not find container \"8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f\": container with ID starting with 8c869953baaa2d338e7dc92c1bddfc26c7d594c18afd6c66a60156c14a8e4e3f not found: ID does not exist"
Oct 30 21:09:08.448830 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.448791    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"]
Oct 30 21:09:08.452477 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:08.452449    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a064b4-6bc76c8c66-f9vhk"]
Oct 30 21:09:09.096721 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:09.096683    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:09.331691 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:09.331654    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:09.523935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:09.523905    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:09.965738 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:09.965703    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98bdadb-0093-49eb-b21c-92b0c19fb4ac" path="/var/lib/kubelet/pods/c98bdadb-0093-49eb-b21c-92b0c19fb4ac/volumes"
Oct 30 21:09:10.026866 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:10.026834    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:12.855880 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:12.855845    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:09:14.325428 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:14.325384    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:14.926127 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:14.926086    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" podUID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:09:14.926331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:14.926086    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" podUID="d15d64ca-0058-43ef-90a2-b879e816a641" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:09:16.208072 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:16.208012    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:16.405768 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:16.405730    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:16.624549 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:16.624489    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:17.216341 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:17.216298    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:19.314936 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:19.314896    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:22.102592 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:22.102554    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"]
Oct 30 21:09:22.109383 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:22.109352    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"]
Oct 30 21:09:22.264508 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:22.264471    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:23.612577 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.612489    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:09:23.688397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.688369    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15d64ca-0058-43ef-90a2-b879e816a641-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15d64ca-0058-43ef-90a2-b879e816a641-plugins-config-volume\") pod \"d15d64ca-0058-43ef-90a2-b879e816a641\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") "
Oct 30 21:09:23.688585 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.688414    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15d64ca-0058-43ef-90a2-b879e816a641-kube-api-access-22jfk\" (UniqueName: \"kubernetes.io/projected/d15d64ca-0058-43ef-90a2-b879e816a641-kube-api-access-22jfk\") pod \"d15d64ca-0058-43ef-90a2-b879e816a641\" (UID: \"d15d64ca-0058-43ef-90a2-b879e816a641\") "
Oct 30 21:09:23.688800 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.688775    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15d64ca-0058-43ef-90a2-b879e816a641-plugins-config-volume" pod "d15d64ca-0058-43ef-90a2-b879e816a641" (UID: "d15d64ca-0058-43ef-90a2-b879e816a641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:23.688909 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.688883    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15d64ca-0058-43ef-90a2-b879e816a641-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:23.690718 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.690694    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15d64ca-0058-43ef-90a2-b879e816a641-kube-api-access-22jfk" pod "d15d64ca-0058-43ef-90a2-b879e816a641" (UID: "d15d64ca-0058-43ef-90a2-b879e816a641"). InnerVolumeSpecName "kube-api-access-22jfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:23.703176 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.703151    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:09:23.790212 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.790172    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-kube-api-access-qf2sg\" (UniqueName: \"kubernetes.io/projected/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-kube-api-access-qf2sg\") pod \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") "
Oct 30 21:09:23.790212 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.790215    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-plugins-config-volume\") pod \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\" (UID: \"41a0ad7b-a3c5-496f-b75d-b295d8733fd6\") "
Oct 30 21:09:23.790574 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.790524    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jfk\" (UniqueName: \"kubernetes.io/projected/d15d64ca-0058-43ef-90a2-b879e816a641-kube-api-access-22jfk\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:23.790686 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.790667    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-plugins-config-volume" pod "41a0ad7b-a3c5-496f-b75d-b295d8733fd6" (UID: "41a0ad7b-a3c5-496f-b75d-b295d8733fd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:23.792489 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.792466    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-kube-api-access-qf2sg" pod "41a0ad7b-a3c5-496f-b75d-b295d8733fd6" (UID: "41a0ad7b-a3c5-496f-b75d-b295d8733fd6"). InnerVolumeSpecName "kube-api-access-qf2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:23.891338 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.891256    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:23.891338 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:23.891284    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf2sg\" (UniqueName: \"kubernetes.io/projected/41a0ad7b-a3c5-496f-b75d-b295d8733fd6-kube-api-access-qf2sg\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:24.313787 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.313748    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:24.375921 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.375886    5002 generic.go:356] "Generic (PLEG): container finished" podID="d15d64ca-0058-43ef-90a2-b879e816a641" containerID="54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7" exitCode=0
Oct 30 21:09:24.376106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.375934    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"
Oct 30 21:09:24.376106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.375932    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" event={"ID":"d15d64ca-0058-43ef-90a2-b879e816a641","Type":"ContainerDied","Data":"54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7"}
Oct 30 21:09:24.376106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.375993    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm" event={"ID":"d15d64ca-0058-43ef-90a2-b879e816a641","Type":"ContainerDied","Data":"0913252ff9f4c7775ac3294b0097cec7c8bbdee08cef10489fc05e8e021eab13"}
Oct 30 21:09:24.376106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.376015    5002 scope.go:122] "RemoveContainer" containerID="54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7"
Oct 30 21:09:24.377851 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.377823    5002 generic.go:356] "Generic (PLEG): container finished" podID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" containerID="e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d" exitCode=0
Oct 30 21:09:24.377989 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.377870    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"
Oct 30 21:09:24.377989 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.377915    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" event={"ID":"41a0ad7b-a3c5-496f-b75d-b295d8733fd6","Type":"ContainerDied","Data":"e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d"}
Oct 30 21:09:24.377989 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.377963    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw" event={"ID":"41a0ad7b-a3c5-496f-b75d-b295d8733fd6","Type":"ContainerDied","Data":"91ee9130acf58d2fd44837c4b1bf38884d44a6ff91edfd4a0bfbef70d8f195ac"}
Oct 30 21:09:24.385692 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.385669    5002 scope.go:122] "RemoveContainer" containerID="54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7"
Oct 30 21:09:24.385972 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:09:24.385949    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7\": container with ID starting with 54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7 not found: ID does not exist" containerID="54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7"
Oct 30 21:09:24.386030 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.385986    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7"} err="failed to get container status \"54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7\": rpc error: code = NotFound desc = could not find container \"54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7\": container with ID starting with 54430d95455a6470ffe74f23656cd80b13b57abd737d7bdc3e802dd5a30f79a7 not found: ID does not exist"
Oct 30 21:09:24.386030 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.386009    5002 scope.go:122] "RemoveContainer" containerID="e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d"
Oct 30 21:09:24.395113 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.395091    5002 scope.go:122] "RemoveContainer" containerID="e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d"
Oct 30 21:09:24.395431 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:09:24.395413    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d\": container with ID starting with e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d not found: ID does not exist" containerID="e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d"
Oct 30 21:09:24.395492 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.395442    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d"} err="failed to get container status \"e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d\": rpc error: code = NotFound desc = could not find container \"e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d\": container with ID starting with e1721343b9ded06da9b896aea3c58f1142ab23290f01524ccf8f53ee53c3c07d not found: ID does not exist"
Oct 30 21:09:24.541794 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.541756    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"]
Oct 30 21:09:24.545329 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.545291    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm"]
Oct 30 21:09:24.657148 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.657060    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"]
Oct 30 21:09:24.660624 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:24.660595    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw"]
Oct 30 21:09:25.965796 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:25.965760    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" path="/var/lib/kubelet/pods/41a0ad7b-a3c5-496f-b75d-b295d8733fd6/volumes"
Oct 30 21:09:25.966205 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:25.966147    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15d64ca-0058-43ef-90a2-b879e816a641" path="/var/lib/kubelet/pods/d15d64ca-0058-43ef-90a2-b879e816a641/volumes"
Oct 30 21:09:28.984214 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:28.984175    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:29.359659 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:29.359562    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"]
Oct 30 21:09:29.657950 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:29.657862    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:29.833616 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:29.833584    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:33.135386 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135346    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"]
Oct 30 21:09:33.135846 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135824    5002 cpu_manager.go:414] "containerMap: removing container" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy"
Oct 30 21:09:33.135846 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135836    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy"
Oct 30 21:09:33.135846 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135846    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d15d64ca-0058-43ef-90a2-b879e816a641" containerName="payload-processing"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135851    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15d64ca-0058-43ef-90a2-b879e816a641" containerName="payload-processing"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135861    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c98bdadb-0093-49eb-b21c-92b0c19fb4ac" containerName="maas-api"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135867    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98bdadb-0093-49eb-b21c-92b0c19fb4ac" containerName="maas-api"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135881    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c784da18-e7c2-4288-b725-7ca1b42450e2" containerName="payload-processing"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135886    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c784da18-e7c2-4288-b725-7ca1b42450e2" containerName="payload-processing"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135893    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9da76b40-3caf-4cc2-99db-63cf46d612b7" containerName="main"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135899    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da76b40-3caf-4cc2-99db-63cf46d612b7" containerName="main"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135909    5002 cpu_manager.go:414] "containerMap: removing container" podUID="991c5524-1638-4f50-ab27-289d7ab6824a" containerName="maas-api"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135913    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991c5524-1638-4f50-ab27-289d7ab6824a" containerName="maas-api"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135919    5002 cpu_manager.go:414] "containerMap: removing container" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135924    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135929    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135934    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135942    5002 cpu_manager.go:414] "containerMap: removing container" podUID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" containerName="payload-pre-processing"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135947    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" containerName="payload-pre-processing"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135959    5002 cpu_manager.go:414] "containerMap: removing container" podUID="114a89a2-44c3-4622-90a3-827fd3651162" containerName="maas-api"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135964    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114a89a2-44c3-4622-90a3-827fd3651162" containerName="maas-api"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135972    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy"
Oct 30 21:09:33.135971 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135977    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135985    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.135990    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136000    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e430c6d2-88ee-413f-8b7d-f90efe62a92f" containerName="maas-api"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136006    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e430c6d2-88ee-413f-8b7d-f90efe62a92f" containerName="maas-api"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136019    5002 cpu_manager.go:414] "containerMap: removing container" podUID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" containerName="payload-pre-processing"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136023    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" containerName="payload-pre-processing"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136084    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9725de7-41ec-4ebd-bde8-f9fe5c741209" containerName="payload-pre-processing"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136094    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="114a89a2-44c3-4622-90a3-827fd3651162" containerName="maas-api"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136101    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1286f6f-71b5-499b-80ae-aafcf82c9c8a" containerName="istio-proxy"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136106    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136111    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e430c6d2-88ee-413f-8b7d-f90efe62a92f" containerName="maas-api"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136117    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a0ad7b-a3c5-496f-b75d-b295d8733fd6" containerName="payload-pre-processing"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136123    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136127    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15d64ca-0058-43ef-90a2-b879e816a641" containerName="payload-processing"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136132    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98bdadb-0093-49eb-b21c-92b0c19fb4ac" containerName="maas-api"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136138    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="c784da18-e7c2-4288-b725-7ca1b42450e2" containerName="payload-processing"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136142    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da76b40-3caf-4cc2-99db-63cf46d612b7" containerName="main"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136147    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="991c5524-1638-4f50-ab27-289d7ab6824a" containerName="maas-api"
Oct 30 21:09:33.136741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.136152    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b5bd60-b2cb-4fbb-8b5b-f93f166dd64f" containerName="istio-proxy"
Oct 30 21:09:33.139213 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.139195    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.185925 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.185884    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqndt\" (UniqueName: \"kubernetes.io/projected/8f1f1162-4dc9-4967-b098-a2ef09c96612-kube-api-access-cqndt\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.186128 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.185953    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.186128 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.185995    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.197219 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.197186    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"]
Oct 30 21:09:33.244185 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.244147    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:09:33.244185 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.244159    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:09:33.244414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.244192    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:09:33.244414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.244222    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:09:33.287155 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.287112    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqndt\" (UniqueName: \"kubernetes.io/projected/8f1f1162-4dc9-4967-b098-a2ef09c96612-kube-api-access-cqndt\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.287351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.287170    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.287351 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.287193    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.287821 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.287793    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.289808 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.289780    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.344807 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.344768    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqndt\" (UniqueName: \"kubernetes.io/projected/8f1f1162-4dc9-4967-b098-a2ef09c96612-kube-api-access-cqndt\") pod \"maas-api-revoke-sub-44cbd5c3-mtcxn\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.455565 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.455439    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:33.832404 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:33.832370    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"]
Oct 30 21:09:33.833905 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:09:33.833876    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1f1162_4dc9_4967_b098_a2ef09c96612.slice/crio-5529256f1daff185b49364d13d05cfcee4bc9e191f3554e02499c3810520e53f WatchSource:0}: Error finding container 5529256f1daff185b49364d13d05cfcee4bc9e191f3554e02499c3810520e53f: Status 404 returned error can't find the container with id 5529256f1daff185b49364d13d05cfcee4bc9e191f3554e02499c3810520e53f
Oct 30 21:09:34.360653 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.360603    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg" podUID="bc9fa523-13bb-4f20-9410-b4e49f214dd1" containerName="payload-pre-processing" containerID="cri-o://f3734afcbee7ece60bc41d19023a77c752f1e418da8782b52414e03717bdf37f" gracePeriod=25
Oct 30 21:09:34.447084 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.447047    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn" event={"ID":"8f1f1162-4dc9-4967-b098-a2ef09c96612","Type":"ContainerStarted","Data":"145e9b7b3d691c9ad8f69dba21969ea7e188a302e816aaab40b8545383d68a70"}
Oct 30 21:09:34.447303 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.447143    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn" event={"ID":"8f1f1162-4dc9-4967-b098-a2ef09c96612","Type":"ContainerStarted","Data":"5529256f1daff185b49364d13d05cfcee4bc9e191f3554e02499c3810520e53f"}
Oct 30 21:09:34.455127 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.455097    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:09:34.471820 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.471791    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:09:34.564986 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.564911    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn" podStartSLOduration=1.564882076 podStartE2EDuration="1.564882076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:34.562819975 +0000 UTC m=+3000.711870963" watchObservedRunningTime="2026-10-30 21:09:34.564882076 +0000 UTC m=+3000.713933065"
Oct 30 21:09:34.634152 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.634077    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 21:09:34.701219 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.701187    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9fa523-13bb-4f20-9410-b4e49f214dd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9fa523-13bb-4f20-9410-b4e49f214dd1-plugins-config-volume\") pod \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") "
Oct 30 21:09:34.701420 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.701301    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9fa523-13bb-4f20-9410-b4e49f214dd1-kube-api-access-k6vkj\" (UniqueName: \"kubernetes.io/projected/bc9fa523-13bb-4f20-9410-b4e49f214dd1-kube-api-access-k6vkj\") pod \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\" (UID: \"bc9fa523-13bb-4f20-9410-b4e49f214dd1\") "
Oct 30 21:09:34.701640 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.701614    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9fa523-13bb-4f20-9410-b4e49f214dd1-plugins-config-volume" pod "bc9fa523-13bb-4f20-9410-b4e49f214dd1" (UID: "bc9fa523-13bb-4f20-9410-b4e49f214dd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:34.703568 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.703542    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9fa523-13bb-4f20-9410-b4e49f214dd1-kube-api-access-k6vkj" pod "bc9fa523-13bb-4f20-9410-b4e49f214dd1" (UID: "bc9fa523-13bb-4f20-9410-b4e49f214dd1"). InnerVolumeSpecName "kube-api-access-k6vkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:34.802736 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.802694    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vkj\" (UniqueName: \"kubernetes.io/projected/bc9fa523-13bb-4f20-9410-b4e49f214dd1-kube-api-access-k6vkj\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:34.802736 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:34.802738    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9fa523-13bb-4f20-9410-b4e49f214dd1-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:35.118560 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.118506    5002 scope.go:122] "RemoveContainer" containerID="f3734afcbee7ece60bc41d19023a77c752f1e418da8782b52414e03717bdf37f"
Oct 30 21:09:35.454138 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.454041    5002 generic.go:356] "Generic (PLEG): container finished" podID="8f1f1162-4dc9-4967-b098-a2ef09c96612" containerID="145e9b7b3d691c9ad8f69dba21969ea7e188a302e816aaab40b8545383d68a70" exitCode=0
Oct 30 21:09:35.454645 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.454132    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn" event={"ID":"8f1f1162-4dc9-4967-b098-a2ef09c96612","Type":"ContainerDied","Data":"145e9b7b3d691c9ad8f69dba21969ea7e188a302e816aaab40b8545383d68a70"}
Oct 30 21:09:35.454645 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.454281    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"
Oct 30 21:09:35.454645 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.454285    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg" event={"ID":"bc9fa523-13bb-4f20-9410-b4e49f214dd1","Type":"ContainerDied","Data":"f3734afcbee7ece60bc41d19023a77c752f1e418da8782b52414e03717bdf37f"}
Oct 30 21:09:35.454645 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.454319    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg" event={"ID":"bc9fa523-13bb-4f20-9410-b4e49f214dd1","Type":"ContainerDied","Data":"d7bce760c292f67d72543e843ceaee2ab0e5fc12af07870923f19405221c8b59"}
Oct 30 21:09:35.749354 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.749307    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"]
Oct 30 21:09:35.753288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.753249    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg"]
Oct 30 21:09:35.967399 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:35.967363    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9fa523-13bb-4f20-9410-b4e49f214dd1" path="/var/lib/kubelet/pods/bc9fa523-13bb-4f20-9410-b4e49f214dd1/volumes"
Oct 30 21:09:36.611124 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.611093    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:36.620807 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.620774    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-openshift-ca\") pod \"8f1f1162-4dc9-4967-b098-a2ef09c96612\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") "
Oct 30 21:09:36.621038 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.620848    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1f1162-4dc9-4967-b098-a2ef09c96612-kube-api-access-cqndt\" (UniqueName: \"kubernetes.io/projected/8f1f1162-4dc9-4967-b098-a2ef09c96612-kube-api-access-cqndt\") pod \"8f1f1162-4dc9-4967-b098-a2ef09c96612\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") "
Oct 30 21:09:36.621038 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.620902    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-xks-ca\") pod \"8f1f1162-4dc9-4967-b098-a2ef09c96612\" (UID: \"8f1f1162-4dc9-4967-b098-a2ef09c96612\") "
Oct 30 21:09:36.621251 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.621224    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-openshift-ca" pod "8f1f1162-4dc9-4967-b098-a2ef09c96612" (UID: "8f1f1162-4dc9-4967-b098-a2ef09c96612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:36.623300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.623268    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-xks-ca" pod "8f1f1162-4dc9-4967-b098-a2ef09c96612" (UID: "8f1f1162-4dc9-4967-b098-a2ef09c96612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:36.623435 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.623384    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1f1162-4dc9-4967-b098-a2ef09c96612-kube-api-access-cqndt" pod "8f1f1162-4dc9-4967-b098-a2ef09c96612" (UID: "8f1f1162-4dc9-4967-b098-a2ef09c96612"). InnerVolumeSpecName "kube-api-access-cqndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:36.722592 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.722475    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-openshift-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:36.722592 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.722513    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqndt\" (UniqueName: \"kubernetes.io/projected/8f1f1162-4dc9-4967-b098-a2ef09c96612-kube-api-access-cqndt\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:36.722592 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:36.722527    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1f1162-4dc9-4967-b098-a2ef09c96612-maas-api-xks-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.117154 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:09:37.117065    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:09:37.131323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.131285    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main" containerID="cri-o://6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2" gracePeriod=45
Oct 30 21:09:37.132582 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.132509    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main" containerID="cri-o://b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c" gracePeriod=45
Oct 30 21:09:37.402093 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.402063    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 21:09:37.426326 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.426298    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 21:09:37.429470 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429448    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-tmp-dir\") pod \"9b34596f-6026-447b-88a8-8e61ebefaede\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") "
Oct 30 21:09:37.429546 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429515    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-dshm\") pod \"9b34596f-6026-447b-88a8-8e61ebefaede\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") "
Oct 30 21:09:37.429600 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429554    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-home\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-home\") pod \"9b34596f-6026-447b-88a8-8e61ebefaede\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") "
Oct 30 21:09:37.429600 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429581    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-model-cache\") pod \"9b34596f-6026-447b-88a8-8e61ebefaede\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") "
Oct 30 21:09:37.429701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429615    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b34596f-6026-447b-88a8-8e61ebefaede-kube-api-access-2vm9m\" (UniqueName: \"kubernetes.io/projected/9b34596f-6026-447b-88a8-8e61ebefaede-kube-api-access-2vm9m\") pod \"9b34596f-6026-447b-88a8-8e61ebefaede\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") "
Oct 30 21:09:37.429751 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429703    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b34596f-6026-447b-88a8-8e61ebefaede-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b34596f-6026-447b-88a8-8e61ebefaede-tls-certs\") pod \"9b34596f-6026-447b-88a8-8e61ebefaede\" (UID: \"9b34596f-6026-447b-88a8-8e61ebefaede\") "
Oct 30 21:09:37.429751 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429729    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-tmp-dir" pod "9b34596f-6026-447b-88a8-8e61ebefaede" (UID: "9b34596f-6026-447b-88a8-8e61ebefaede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.429928 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429902    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-model-cache" pod "9b34596f-6026-447b-88a8-8e61ebefaede" (UID: "9b34596f-6026-447b-88a8-8e61ebefaede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.429998 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.429906    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-home" pod "9b34596f-6026-447b-88a8-8e61ebefaede" (UID: "9b34596f-6026-447b-88a8-8e61ebefaede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.430041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.430024    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.430084 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.430043    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.430084 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.430057    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.432110 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.432076    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-dshm" pod "9b34596f-6026-447b-88a8-8e61ebefaede" (UID: "9b34596f-6026-447b-88a8-8e61ebefaede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.432221 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.432149    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b34596f-6026-447b-88a8-8e61ebefaede-tls-certs" pod "9b34596f-6026-447b-88a8-8e61ebefaede" (UID: "9b34596f-6026-447b-88a8-8e61ebefaede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:37.432221 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.432177    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b34596f-6026-447b-88a8-8e61ebefaede-kube-api-access-2vm9m" pod "9b34596f-6026-447b-88a8-8e61ebefaede" (UID: "9b34596f-6026-447b-88a8-8e61ebefaede"). InnerVolumeSpecName "kube-api-access-2vm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:37.474995 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.474954    5002 generic.go:356] "Generic (PLEG): container finished" podID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerID="b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c" exitCode=0
Oct 30 21:09:37.475191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.475027    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"
Oct 30 21:09:37.475191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.475028    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" event={"ID":"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6","Type":"ContainerDied","Data":"b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c"}
Oct 30 21:09:37.475191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.475075    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7" event={"ID":"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6","Type":"ContainerDied","Data":"85bf86864908b529dba9cb9c1f910ba4170f5ea547a2dc062429dda73011584a"}
Oct 30 21:09:37.475191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.475092    5002 scope.go:122] "RemoveContainer" containerID="b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c"
Oct 30 21:09:37.476615 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.476591    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn" event={"ID":"8f1f1162-4dc9-4967-b098-a2ef09c96612","Type":"ContainerDied","Data":"5529256f1daff185b49364d13d05cfcee4bc9e191f3554e02499c3810520e53f"}
Oct 30 21:09:37.476684 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.476671    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5529256f1daff185b49364d13d05cfcee4bc9e191f3554e02499c3810520e53f"
Oct 30 21:09:37.476775 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.476762    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"
Oct 30 21:09:37.479031 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.479002    5002 generic.go:356] "Generic (PLEG): container finished" podID="9b34596f-6026-447b-88a8-8e61ebefaede" containerID="6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2" exitCode=0
Oct 30 21:09:37.479175 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.479033    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" event={"ID":"9b34596f-6026-447b-88a8-8e61ebefaede","Type":"ContainerDied","Data":"6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2"}
Oct 30 21:09:37.479175 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.479090    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"
Oct 30 21:09:37.479283 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.479099    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss" event={"ID":"9b34596f-6026-447b-88a8-8e61ebefaede","Type":"ContainerDied","Data":"db955a1be9392d7a213ae9599ad5713d50b6aff03bf62c7ad36fd266126fd354"}
Oct 30 21:09:37.492499 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.492477    5002 scope.go:122] "RemoveContainer" containerID="b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c"
Oct 30 21:09:37.494497 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:09:37.494401    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c\": container with ID starting with b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c not found: ID does not exist" containerID="b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c"
Oct 30 21:09:37.494497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.494448    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c"} err="failed to get container status \"b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c\": rpc error: code = NotFound desc = could not find container \"b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c\": container with ID starting with b464fa6ba941d34d6917ae0eb64f9537d3b322a223336604fc9170ece946565c not found: ID does not exist"
Oct 30 21:09:37.494678 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.494478    5002 scope.go:122] "RemoveContainer" containerID="6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2"
Oct 30 21:09:37.505973 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.505948    5002 scope.go:122] "RemoveContainer" containerID="6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2"
Oct 30 21:09:37.506337 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:09:37.506312    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2\": container with ID starting with 6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2 not found: ID does not exist" containerID="6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2"
Oct 30 21:09:37.506421 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.506348    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2"} err="failed to get container status \"6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2\": rpc error: code = NotFound desc = could not find container \"6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2\": container with ID starting with 6237a1a05e2b5d5e9a622219d0b4962b05c67f081f7f949e6b998cac545529e2 not found: ID does not exist"
Oct 30 21:09:37.530577 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.530514    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-dshm\") pod \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") "
Oct 30 21:09:37.530777 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.530651    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-kube-api-access-f5fv6\" (UniqueName: \"kubernetes.io/projected/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-kube-api-access-f5fv6\") pod \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") "
Oct 30 21:09:37.530777 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.530675    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-model-cache\") pod \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") "
Oct 30 21:09:37.530777 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.530705    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tls-certs\") pod \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") "
Oct 30 21:09:37.530942 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.530859    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tmp-dir\") pod \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") "
Oct 30 21:09:37.530942 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.530891    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-home\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-home\") pod \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\" (UID: \"d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6\") "
Oct 30 21:09:37.531120 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.531096    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-model-cache" pod "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" (UID: "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.531296 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.531279    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.531361 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.531304    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b34596f-6026-447b-88a8-8e61ebefaede-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.531361 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.531319    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b34596f-6026-447b-88a8-8e61ebefaede-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.531361 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.531334    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vm9m\" (UniqueName: \"kubernetes.io/projected/9b34596f-6026-447b-88a8-8e61ebefaede-kube-api-access-2vm9m\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.531361 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.531301    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-home" pod "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" (UID: "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.534325 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.531605    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tmp-dir" pod "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" (UID: "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.534622 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.534573    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-dshm" pod "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" (UID: "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.534746 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.534704    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tls-certs" pod "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" (UID: "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:37.534746 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.534710    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-kube-api-access-f5fv6" pod "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" (UID: "d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6"). InnerVolumeSpecName "kube-api-access-f5fv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:37.632384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.632350    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5fv6\" (UniqueName: \"kubernetes.io/projected/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-kube-api-access-f5fv6\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.632384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.632382    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.632850 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.632413    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.632850 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.632422    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.632850 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.632430    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.820062 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.820018    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"]
Oct 30 21:09:37.822233 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.822211    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-w6-4ee72d-kserve-696686446b-k46ss"]
Oct 30 21:09:37.935840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.935809    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"]
Oct 30 21:09:37.947054 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.947021    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/premium-w6-4ee72d-kserve-6ff6b6898c-t5jm7"]
Oct 30 21:09:37.966462 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.966426    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" path="/var/lib/kubelet/pods/9b34596f-6026-447b-88a8-8e61ebefaede/volumes"
Oct 30 21:09:37.966958 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:37.966938    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" path="/var/lib/kubelet/pods/d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6/volumes"
Oct 30 21:09:41.144214 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:41.144183    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:48.509108 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:48.509023    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:50.023935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:50.023889    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:09:50.094159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:50.094125    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:50.365775 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:50.365651    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:51.014667 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:51.014635    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:57.647043 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:57.646997    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:57.850635 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:57.850594    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:58.065179 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:58.065138    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:58.451428 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:58.451330    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:58.629479 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:58.629445    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:59.550481 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:09:59.550443    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:05.020754 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:05.020720    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:05.132386 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:05.132349    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:10:07.121039 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:10:07.120958    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:10:18.200658 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:18.200616    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:33.941307 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:33.941270    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:10:37.118228 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:10:37.118178    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:10:37.560826 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.560788    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"]
Oct 30 21:10:37.561239 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561226    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8f1f1162-4dc9-4967-b098-a2ef09c96612" containerName="revoke-keys"
Oct 30 21:10:37.561295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561241    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1f1162-4dc9-4967-b098-a2ef09c96612" containerName="revoke-keys"
Oct 30 21:10:37.561295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561261    5002 cpu_manager.go:414] "containerMap: removing container" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main"
Oct 30 21:10:37.561295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561266    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main"
Oct 30 21:10:37.561295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561274    5002 cpu_manager.go:414] "containerMap: removing container" podUID="bc9fa523-13bb-4f20-9410-b4e49f214dd1" containerName="payload-pre-processing"
Oct 30 21:10:37.561295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561279    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9fa523-13bb-4f20-9410-b4e49f214dd1" containerName="payload-pre-processing"
Oct 30 21:10:37.561295 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561295    5002 cpu_manager.go:414] "containerMap: removing container" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main"
Oct 30 21:10:37.561517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561299    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main"
Oct 30 21:10:37.561517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561359    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12f1a1f-6ebd-48c6-9da5-b47e2c8a5ff6" containerName="main"
Oct 30 21:10:37.561517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561369    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1f1162-4dc9-4967-b098-a2ef09c96612" containerName="revoke-keys"
Oct 30 21:10:37.561517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561381    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89051f8-4d46-4f57-9fb0-66535ddfd473" containerName="istio-proxy"
Oct 30 21:10:37.561517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561388    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b34596f-6026-447b-88a8-8e61ebefaede" containerName="main"
Oct 30 21:10:37.561517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.561394    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9fa523-13bb-4f20-9410-b4e49f214dd1" containerName="payload-pre-processing"
Oct 30 21:10:37.565891 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.565870    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.611331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.611285    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"]
Oct 30 21:10:37.615092 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.615065    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.626843 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.626818    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.626995 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.626852    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.626995 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.626890    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.627100 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.626989    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.627139 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.627098    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.627139 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.627123    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.627207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.627148    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xql24\" (UniqueName: \"kubernetes.io/projected/c77cbfe3-cbd1-48c7-8685-9745a82600dd-kube-api-access-xql24\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.627207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.627200    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.627272 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.627250    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.668816 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.668778    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"]
Oct 30 21:10:37.721126 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.721089    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"]
Oct 30 21:10:37.728102 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728057    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728112    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728151    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728226    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728280    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728315    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728354    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728402    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4xf\" (UniqueName: \"kubernetes.io/projected/13afaa6b-b0a8-42ef-8149-33c57717f89e-kube-api-access-4b4xf\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728438    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728469    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13afaa6b-b0a8-42ef-8149-33c57717f89e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728511    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728563    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728627    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728679    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728708    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728753    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xql24\" (UniqueName: \"kubernetes.io/projected/c77cbfe3-cbd1-48c7-8685-9745a82600dd-kube-api-access-xql24\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728784    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728823    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.728867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.728826    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.729321 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.729086    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.729383 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.729351    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.729742 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.729714    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.730014 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.729989    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.732285 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.732263    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.734953 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.734929    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.787466 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.787431    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xql24\" (UniqueName: \"kubernetes.io/projected/c77cbfe3-cbd1-48c7-8685-9745a82600dd-kube-api-access-xql24\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.791049 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.791019    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77cbfe3-cbd1-48c7-8685-9745a82600dd-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-tjm7q\" (UID: \"c77cbfe3-cbd1-48c7-8685-9745a82600dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.829762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.829659    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.829762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.829717    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.829762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.829745    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4xf\" (UniqueName: \"kubernetes.io/projected/13afaa6b-b0a8-42ef-8149-33c57717f89e-kube-api-access-4b4xf\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.829769    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13afaa6b-b0a8-42ef-8149-33c57717f89e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.829848    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.829912    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.829977    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.830017    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.830053    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830421 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.830154    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830421 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.830339    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830421 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.830396    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830616 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.830592    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.830699 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.830680    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13afaa6b-b0a8-42ef-8149-33c57717f89e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.833225 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.833191    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.833752 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.833724    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.879683 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.879644    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:37.892260 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.892219    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13afaa6b-b0a8-42ef-8149-33c57717f89e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.892433 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.892304    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4xf\" (UniqueName: \"kubernetes.io/projected/13afaa6b-b0a8-42ef-8149-33c57717f89e-kube-api-access-4b4xf\") pod \"maas-default-gateway-openshift-default-6886b96f9-sf6r9\" (UID: \"13afaa6b-b0a8-42ef-8149-33c57717f89e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:37.928796 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:37.928754    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:38.316443 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.316406    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"]
Oct 30 21:10:38.354598 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:10:38.354558    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77cbfe3_cbd1_48c7_8685_9745a82600dd.slice/crio-a5342d27287d7cd84d0c758a38836f7b644723f0f3d3589f58e4776012a36c19 WatchSource:0}: Error finding container a5342d27287d7cd84d0c758a38836f7b644723f0f3d3589f58e4776012a36c19: Status 404 returned error can't find the container with id a5342d27287d7cd84d0c758a38836f7b644723f0f3d3589f58e4776012a36c19
Oct 30 21:10:38.356786 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.356750    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.356873 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.356847    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.356924 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.356892    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.377778 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.377749    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"]
Oct 30 21:10:38.390765 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:10:38.390739    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13afaa6b_b0a8_42ef_8149_33c57717f89e.slice/crio-bfbacfa96d55864ce024c0f26d20f724d248a61cb1083e984c35bcf8ed19bbe2 WatchSource:0}: Error finding container bfbacfa96d55864ce024c0f26d20f724d248a61cb1083e984c35bcf8ed19bbe2: Status 404 returned error can't find the container with id bfbacfa96d55864ce024c0f26d20f724d248a61cb1083e984c35bcf8ed19bbe2
Oct 30 21:10:38.393288 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.393255    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.393373 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.393347    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.393414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.393382    5002 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.860213 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.860164    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" event={"ID":"13afaa6b-b0a8-42ef-8149-33c57717f89e","Type":"ContainerStarted","Data":"c5127e0a1ca2d5baf335345860d2ea5bdeb452e32d583fb21db56d0ac37820c9"}
Oct 30 21:10:38.860554 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.860509    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" event={"ID":"13afaa6b-b0a8-42ef-8149-33c57717f89e","Type":"ContainerStarted","Data":"bfbacfa96d55864ce024c0f26d20f724d248a61cb1083e984c35bcf8ed19bbe2"}
Oct 30 21:10:38.860698 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.860676    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" event={"ID":"c77cbfe3-cbd1-48c7-8685-9745a82600dd","Type":"ContainerStarted","Data":"6a3505749182ae65b3d700dd8ae3be2ea9911e6d095434e4f188ab437c0b5b7c"}
Oct 30 21:10:38.860821 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.860807    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" event={"ID":"c77cbfe3-cbd1-48c7-8685-9745a82600dd","Type":"ContainerStarted","Data":"a5342d27287d7cd84d0c758a38836f7b644723f0f3d3589f58e4776012a36c19"}
Oct 30 21:10:38.879901 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.879861    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:38.929674 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.929635    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:38.984050 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:38.983992    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podStartSLOduration=1.983977866 podStartE2EDuration="1.983977866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:38.981477082 +0000 UTC m=+3065.130528071" watchObservedRunningTime="2026-10-30 21:10:38.983977866 +0000 UTC m=+3065.133028858"
Oct 30 21:10:39.104013 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:39.103953    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podStartSLOduration=2.103938268 podStartE2EDuration="2.103938268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:39.103030665 +0000 UTC m=+3065.252081653" watchObservedRunningTime="2026-10-30 21:10:39.103938268 +0000 UTC m=+3065.252989255"
Oct 30 21:10:39.882416 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:39.882381    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:10:39.883109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:39.882438    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 21:10:39.935388 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:39.935346    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:10:39.935716 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:39.935683    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 21:10:40.880510 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:40.880474    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:10:40.880743 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:40.880523    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 21:10:40.930559 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:40.929953    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:10:40.930559 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:40.930018    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 21:10:41.452560 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:41.451724    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:41.881384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:41.881349    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:10:41.881635 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:41.881408    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 21:10:41.945802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:41.945767    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:10:41.946370 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:41.946344    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 21:10:42.524577 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:42.514627    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:42.880844 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:42.880758    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:10:42.880844 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:42.880813    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 21:10:42.933559 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:42.932969    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:10:42.933559 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:42.933038    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 21:10:43.881105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:43.881060    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:10:43.881572 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:43.881131    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 21:10:43.930126 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:43.930089    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:10:43.930449 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:43.930415    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 21:10:44.602158 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:44.602119    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:44.880451 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:44.880333    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:10:44.880451 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:44.880407    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 21:10:44.930059 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:44.930021    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:10:44.930648 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:44.930078    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 21:10:45.880329 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:45.880285    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:10:45.880619 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:45.880355    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 21:10:45.929806 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:45.929759    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:10:45.929998 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:45.929837    5002 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 21:10:46.887779 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:46.887711    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:46.888281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:46.888257    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:46.889191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:46.889165    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q"
Oct 30 21:10:46.933154 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:46.933125    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:46.933508 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:46.933488    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:10:46.934230 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:10:46.934211    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9"
Oct 30 21:11:07.115567 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:11:07.115508    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:11:29.077091 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:29.077053    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:29.845464 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:29.845426    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:11:32.320559 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:32.319677    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:32.320559 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:32.319783    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7" podUID="ca7680e5-2ded-4caf-a212-2863026c6641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:32.897862 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:32.897804    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:32.898210 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:32.898164    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q" podUID="c77cbfe3-cbd1-48c7-8685-9745a82600dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:32.996053 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:32.995774    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:32.996389 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:32.996339    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9" podUID="13afaa6b-b0a8-42ef-8149-33c57717f89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:37.118406 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:11:37.118362    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:11:41.791487 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:41.791450    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:51.311943 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:11:51.311908    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:12:00.602103 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:12:00.602062    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:12:00.767403 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:12:00.767344    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:12:02.451445 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:12:02.451407    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:12:07.116281 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:12:07.116240    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:12:22.951401 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:12:22.951365    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:12:37.115828 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:12:37.115785    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:12:41.552510 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:12:41.552471    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:12:42.136802 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:12:42.136766    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:07.115828 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:13:07.115790    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:13:13.943355 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:13:13.943314    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:13:24.711458 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:13:24.711412    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:13:29.602896 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:13:29.602860    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:29.772728 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:13:29.772692    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:37.118032 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:13:37.117976    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:13:37.538308 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:13:37.538269    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:13:42.488137 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:13:42.488093    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:02.751595 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:02.751549    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:14:07.116791 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:14:07.116748    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:14:10.958517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:10.958478    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:14:31.006882 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:31.006843    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:14:34.493359 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:34.493328    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:14:34.514274 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:34.514246    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:14:37.053631 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:37.053583    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"]
Oct 30 21:14:37.103032 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:37.102992    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44cbd5c3-mtcxn"]
Oct 30 21:14:37.122449 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:14:37.122402    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:14:37.971217 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:37.971170    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1f1162-4dc9-4967-b098-a2ef09c96612" path="/var/lib/kubelet/pods/8f1f1162-4dc9-4967-b098-a2ef09c96612/volumes"
Oct 30 21:14:42.817658 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:42.817613    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:43.966330 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:14:43.966299    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:15:03.618621 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:03.618584    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:15:06.814561 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:06.813680    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:06.814561 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:06.813822    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:07.149692 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:15:07.148738    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:15:12.932924 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:12.932886    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"]
Oct 30 21:15:12.936755 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:12.936735    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:12.990432 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:12.990394    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:13.040892 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.040848    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:13.041125 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.040900    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:13.041125 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.041024    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:15:13.043562 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.043514    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"]
Oct 30 21:15:13.060687 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.060650    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zx46\" (UniqueName: \"kubernetes.io/projected/537924bf-9989-44a0-99ce-1529d8910055-kube-api-access-9zx46\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.060848 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.060700    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537924bf-9989-44a0-99ce-1529d8910055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.060848 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.060727    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537924bf-9989-44a0-99ce-1529d8910055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.161556 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.161499    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx46\" (UniqueName: \"kubernetes.io/projected/537924bf-9989-44a0-99ce-1529d8910055-kube-api-access-9zx46\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.161741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.161590    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537924bf-9989-44a0-99ce-1529d8910055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.161741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.161628    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537924bf-9989-44a0-99ce-1529d8910055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.162398 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.162364    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537924bf-9989-44a0-99ce-1529d8910055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.164175 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.164145    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537924bf-9989-44a0-99ce-1529d8910055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.219497 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.219462    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx46\" (UniqueName: \"kubernetes.io/projected/537924bf-9989-44a0-99ce-1529d8910055-kube-api-access-9zx46\") pod \"maas-api-revoke-sub-0b7e8edb-sbg25\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.250305 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.250268    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:13.626192 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.626159    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"]
Oct 30 21:15:13.627735 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:15:13.627708    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537924bf_9989_44a0_99ce_1529d8910055.slice/crio-49f5ffb9834bd8aaa8c9f048740326a63ff6f34243bff29d0e0205b6935b4542 WatchSource:0}: Error finding container 49f5ffb9834bd8aaa8c9f048740326a63ff6f34243bff29d0e0205b6935b4542: Status 404 returned error can't find the container with id 49f5ffb9834bd8aaa8c9f048740326a63ff6f34243bff29d0e0205b6935b4542
Oct 30 21:15:13.724664 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:13.724624    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25" event={"ID":"537924bf-9989-44a0-99ce-1529d8910055","Type":"ContainerStarted","Data":"49f5ffb9834bd8aaa8c9f048740326a63ff6f34243bff29d0e0205b6935b4542"}
Oct 30 21:15:14.732954 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:14.732870    5002 generic.go:356] "Generic (PLEG): container finished" podID="537924bf-9989-44a0-99ce-1529d8910055" containerID="e2eb24e0107f474d39b16ad88aa36b26b89fab0ab054c3f969dde199d2519a2b" exitCode=0
Oct 30 21:15:14.733337 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:14.732960    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25" event={"ID":"537924bf-9989-44a0-99ce-1529d8910055","Type":"ContainerDied","Data":"e2eb24e0107f474d39b16ad88aa36b26b89fab0ab054c3f969dde199d2519a2b"}
Oct 30 21:15:15.885749 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.885721    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:15.990870 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.990775    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/537924bf-9989-44a0-99ce-1529d8910055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537924bf-9989-44a0-99ce-1529d8910055-maas-api-xks-ca\") pod \"537924bf-9989-44a0-99ce-1529d8910055\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") "
Oct 30 21:15:15.990870 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.990826    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/537924bf-9989-44a0-99ce-1529d8910055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537924bf-9989-44a0-99ce-1529d8910055-maas-api-openshift-ca\") pod \"537924bf-9989-44a0-99ce-1529d8910055\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") "
Oct 30 21:15:15.990870 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.990855    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537924bf-9989-44a0-99ce-1529d8910055-kube-api-access-9zx46\" (UniqueName: \"kubernetes.io/projected/537924bf-9989-44a0-99ce-1529d8910055-kube-api-access-9zx46\") pod \"537924bf-9989-44a0-99ce-1529d8910055\" (UID: \"537924bf-9989-44a0-99ce-1529d8910055\") "
Oct 30 21:15:15.991251 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.991217    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537924bf-9989-44a0-99ce-1529d8910055-maas-api-openshift-ca" pod "537924bf-9989-44a0-99ce-1529d8910055" (UID: "537924bf-9989-44a0-99ce-1529d8910055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:15.991397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.991343    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537924bf-9989-44a0-99ce-1529d8910055-maas-api-openshift-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:15.993148 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.993126    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537924bf-9989-44a0-99ce-1529d8910055-maas-api-xks-ca" pod "537924bf-9989-44a0-99ce-1529d8910055" (UID: "537924bf-9989-44a0-99ce-1529d8910055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:15.993253 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:15.993198    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537924bf-9989-44a0-99ce-1529d8910055-kube-api-access-9zx46" pod "537924bf-9989-44a0-99ce-1529d8910055" (UID: "537924bf-9989-44a0-99ce-1529d8910055"). InnerVolumeSpecName "kube-api-access-9zx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:16.092204 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:16.092160    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537924bf-9989-44a0-99ce-1529d8910055-maas-api-xks-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:16.092204 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:16.092195    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zx46\" (UniqueName: \"kubernetes.io/projected/537924bf-9989-44a0-99ce-1529d8910055-kube-api-access-9zx46\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:16.745834 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:16.745798    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"
Oct 30 21:15:16.746021 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:16.745834    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25" event={"ID":"537924bf-9989-44a0-99ce-1529d8910055","Type":"ContainerDied","Data":"49f5ffb9834bd8aaa8c9f048740326a63ff6f34243bff29d0e0205b6935b4542"}
Oct 30 21:15:16.746021 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:16.745885    5002 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f5ffb9834bd8aaa8c9f048740326a63ff6f34243bff29d0e0205b6935b4542"
Oct 30 21:15:25.814309 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:25.814265    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:15:35.221085 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:35.221047    5002 scope.go:122] "RemoveContainer" containerID="145e9b7b3d691c9ad8f69dba21969ea7e188a302e816aaab40b8545383d68a70"
Oct 30 21:15:35.471762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:35.471675    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:37.116153 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:15:37.116113    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:15:38.939215 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:38.939180    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:15:43.101625 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:43.101513    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:49.762331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:15:49.762293    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:16:07.115583 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:16:07.115543    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:16:09.819104 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:09.819060    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:16:19.850333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:19.850287    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:16:20.394124 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:20.394086    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:34.034489 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:34.034454    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:16:36.809563 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:36.808662    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:36.809563 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:36.808749    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:37.216206 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:16:37.216162    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:16:43.501717 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:43.501678    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:55.923629 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:16:55.923587    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:17:06.614239 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:06.614197    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:17:07.117102 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:17:07.117064    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:17:22.405078 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:22.405036    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:29.062884 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:29.062842    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:17:37.118253 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:17:37.118211    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:17:39.859856 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:39.859819    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:17:43.833868 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:43.833836    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:46.565944 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:46.565911    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:47.182011 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:47.181980    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:47.401714 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:47.401680    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:51.658965 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:51.658925    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:51.916003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:51.915908    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:52.109444 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:52.109411    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:54.958264 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:54.958226    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:55.148292 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.148259    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:55.492895 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.492851    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"]
Oct 30 21:17:55.493819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.493791    5002 cpu_manager.go:414] "containerMap: removing container" podUID="537924bf-9989-44a0-99ce-1529d8910055" containerName="revoke-keys"
Oct 30 21:17:55.493997 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.493983    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="537924bf-9989-44a0-99ce-1529d8910055" containerName="revoke-keys"
Oct 30 21:17:55.494239 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.494226    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="537924bf-9989-44a0-99ce-1529d8910055" containerName="revoke-keys"
Oct 30 21:17:55.500724 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.500700    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.552927 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.552893    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"]
Oct 30 21:17:55.592170 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.592133    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a3165-0de0-43f8-bf64-1f26d29ee8b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.592377 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.592177    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkx2c\" (UniqueName: \"kubernetes.io/projected/694a3165-0de0-43f8-bf64-1f26d29ee8b9-kube-api-access-xkx2c\") pod \"payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.596965 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.596945    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-819d1b\""
Oct 30 21:17:55.597106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.597006    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-95494918-dockercfg-mqcbb\""
Oct 30 21:17:55.662875 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.662840    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"]
Oct 30 21:17:55.666746 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.666715    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:55.693842 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.693810    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a3165-0de0-43f8-bf64-1f26d29ee8b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.693842 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.693852    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkx2c\" (UniqueName: \"kubernetes.io/projected/694a3165-0de0-43f8-bf64-1f26d29ee8b9-kube-api-access-xkx2c\") pod \"payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.694499 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.694476    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a3165-0de0-43f8-bf64-1f26d29ee8b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.724595 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.724561    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"]
Oct 30 21:17:55.751925 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.751826    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkx2c\" (UniqueName: \"kubernetes.io/projected/694a3165-0de0-43f8-bf64-1f26d29ee8b9-kube-api-access-xkx2c\") pod \"payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.794933 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.794894    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc47e4f-3504-4410-9265-31ae3ad03e32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:55.795128 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.794960    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxm5b\" (UniqueName: \"kubernetes.io/projected/5cc47e4f-3504-4410-9265-31ae3ad03e32-kube-api-access-kxm5b\") pod \"payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:55.815581 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.815516    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:55.896506 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.896116    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc47e4f-3504-4410-9265-31ae3ad03e32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:55.896506 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.896181    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxm5b\" (UniqueName: \"kubernetes.io/projected/5cc47e4f-3504-4410-9265-31ae3ad03e32-kube-api-access-kxm5b\") pod \"payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:55.897283 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.897255    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc47e4f-3504-4410-9265-31ae3ad03e32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:55.954918 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.954881    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxm5b\" (UniqueName: \"kubernetes.io/projected/5cc47e4f-3504-4410-9265-31ae3ad03e32-kube-api-access-kxm5b\") pod \"payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:55.979967 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:55.979931    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:56.199895 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.199856    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"]
Oct 30 21:17:56.200791 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:17:56.200764    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694a3165_0de0_43f8_bf64_1f26d29ee8b9.slice/crio-57e81e20975ddbba4d9af01e0ba8c9cac43e05cc0b485ba8ffa65937d2d4c1d6 WatchSource:0}: Error finding container 57e81e20975ddbba4d9af01e0ba8c9cac43e05cc0b485ba8ffa65937d2d4c1d6: Status 404 returned error can't find the container with id 57e81e20975ddbba4d9af01e0ba8c9cac43e05cc0b485ba8ffa65937d2d4c1d6
Oct 30 21:17:56.360840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.360813    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"]
Oct 30 21:17:56.362442 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:17:56.362418    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc47e4f_3504_4410_9265_31ae3ad03e32.slice/crio-06b7904d87eb10246c4479531b578f2ce48529b114ce33826025c02ad1c40c35 WatchSource:0}: Error finding container 06b7904d87eb10246c4479531b578f2ce48529b114ce33826025c02ad1c40c35: Status 404 returned error can't find the container with id 06b7904d87eb10246c4479531b578f2ce48529b114ce33826025c02ad1c40c35
Oct 30 21:17:56.919677 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.919634    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" event={"ID":"694a3165-0de0-43f8-bf64-1f26d29ee8b9","Type":"ContainerStarted","Data":"459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b"}
Oct 30 21:17:56.919912 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.919702    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" event={"ID":"694a3165-0de0-43f8-bf64-1f26d29ee8b9","Type":"ContainerStarted","Data":"57e81e20975ddbba4d9af01e0ba8c9cac43e05cc0b485ba8ffa65937d2d4c1d6"}
Oct 30 21:17:56.919912 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.919802    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:17:56.921602 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.921571    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" event={"ID":"5cc47e4f-3504-4410-9265-31ae3ad03e32","Type":"ContainerStarted","Data":"125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b"}
Oct 30 21:17:56.921748 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.921619    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" event={"ID":"5cc47e4f-3504-4410-9265-31ae3ad03e32","Type":"ContainerStarted","Data":"06b7904d87eb10246c4479531b578f2ce48529b114ce33826025c02ad1c40c35"}
Oct 30 21:17:56.921748 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:56.921642    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:17:57.038175 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.038109    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" podStartSLOduration=2.038092421 podStartE2EDuration="2.038092421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:57.036703358 +0000 UTC m=+3503.185754347" watchObservedRunningTime="2026-10-30 21:17:57.038092421 +0000 UTC m=+3503.187143424"
Oct 30 21:17:57.157987 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.157919    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" podStartSLOduration=2.1578981600000002 podStartE2EDuration="2.15789816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:57.155849978 +0000 UTC m=+3503.304900967" watchObservedRunningTime="2026-10-30 21:17:57.15789816 +0000 UTC m=+3503.306949149"
Oct 30 21:17:57.843855 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.843820    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"]
Oct 30 21:17:57.847512 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.847490    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:57.953024 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.952988    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-819d1b-serving-cert\""
Oct 30 21:17:57.953224 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.953041    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:57.953224 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.953081    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xgxz5\""
Oct 30 21:17:57.953224 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.953086    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:57.953378 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.953266    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:17:57.955799 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:57.955779    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"]
Oct 30 21:17:58.019150 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.019117    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/8ef92562-84d6-4d5e-b1c3-80d79f97100c-kube-api-access-cxb7j\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.019328 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.019171    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92562-84d6-4d5e-b1c3-80d79f97100c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.019328 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.019297    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.019427 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.019406    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.072022 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.071991    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:58.121129 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.121040    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/8ef92562-84d6-4d5e-b1c3-80d79f97100c-kube-api-access-cxb7j\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.121129 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.121082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92562-84d6-4d5e-b1c3-80d79f97100c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.121129 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.121104    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.121370 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.121146    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.121787 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.121755    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92562-84d6-4d5e-b1c3-80d79f97100c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.124061 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.124027    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.124201 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.124027    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.182307 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.182274    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/8ef92562-84d6-4d5e-b1c3-80d79f97100c-kube-api-access-cxb7j\") pod \"maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.254803 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.254764    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:58.432110 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.432027    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:58.463214 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.463183    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:17:58.838237 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.838203    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"]
Oct 30 21:17:58.839120 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:17:58.839096    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef92562_84d6_4d5e_b1c3_80d79f97100c.slice/crio-a0b15a7c45be64bf7c9c37d15d4dc173795c622402151b7d2f8653d329290ac4 WatchSource:0}: Error finding container a0b15a7c45be64bf7c9c37d15d4dc173795c622402151b7d2f8653d329290ac4: Status 404 returned error can't find the container with id a0b15a7c45be64bf7c9c37d15d4dc173795c622402151b7d2f8653d329290ac4
Oct 30 21:17:58.840621 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.840604    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:17:58.937123 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:58.937088    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk" event={"ID":"8ef92562-84d6-4d5e-b1c3-80d79f97100c","Type":"ContainerStarted","Data":"a0b15a7c45be64bf7c9c37d15d4dc173795c622402151b7d2f8653d329290ac4"}
Oct 30 21:17:59.944350 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:59.944256    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk" event={"ID":"8ef92562-84d6-4d5e-b1c3-80d79f97100c","Type":"ContainerStarted","Data":"ea88c2818a0f64e0310867484be6e4f41d7945043e6339acf3d9b01bb001e42f"}
Oct 30 21:17:59.944793 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:17:59.944376    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:18:00.062420 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:00.062360    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk" podStartSLOduration=2.263650738 podStartE2EDuration="3.062345821s" totalImagesPullingTime="798.695083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:00.060620431 +0000 UTC m=+3506.209671411" watchObservedRunningTime="2026-10-30 21:18:00.062345821 +0000 UTC m=+3506.211396809"
Oct 30 21:18:03.155236 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:03.155194    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:05.967507 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:05.967463    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:18:06.153797 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:06.153760    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:18:07.116934 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:18:07.116877    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:18:07.932610 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:07.932575    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:18:07.932851 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:07.932654    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:18:07.949652 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:07.949393    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:12.830833 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:12.830793    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:16.596281 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:16.596238    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:17.243323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:17.243280    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:17.516366 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:17.516262    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:18.085478 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:18.085428    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:20.132812 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:20.132773    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:22.927427 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:22.927369    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:24.657159 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:24.657124    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:18:27.830030 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:27.829899    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:32.662236 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:32.662194    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:37.116349 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:18:37.116310    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:18:37.550864 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:37.550825    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:42.402872 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:42.402788    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:44.173838 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:44.173797    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:47.333516 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:47.333466    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:50.735263 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:50.735222    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:18:52.139847 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:52.139778    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:57.021750 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:18:57.021715    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:01.830255 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:01.830209    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:06.633690 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:06.633647    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:07.116676 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:19:07.116642    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:19:11.453464 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:11.453414    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:16.249075 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:16.249035    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:16.800022 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:16.799984    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:19.842762 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:19.842712    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:19:21.531501 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:21.531470    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:21.822124 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:21.822026    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:26.336574 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:26.336520    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:28.810660 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:28.810616    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"]
Oct 30 21:19:28.920658 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:28.920625    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"]
Oct 30 21:19:29.215502 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:29.215473    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:29.392042 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:29.392006    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:31.238752 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:31.238703    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:33.811646 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:33.811605    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" podUID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" containerName="payload-processing" containerID="cri-o://459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b" gracePeriod=25
Oct 30 21:19:33.921162 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:33.921107    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" podUID="5cc47e4f-3504-4410-9265-31ae3ad03e32" containerName="payload-pre-processing" containerID="cri-o://125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b" gracePeriod=25
Oct 30 21:19:34.530814 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:34.530786    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:19:34.552939 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:34.552908    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:19:35.316680 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:35.316648    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"]
Oct 30 21:19:35.317143 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:35.316858    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk" podUID="8ef92562-84d6-4d5e-b1c3-80d79f97100c" containerName="maas-api" containerID="cri-o://ea88c2818a0f64e0310867484be6e4f41d7945043e6339acf3d9b01bb001e42f" gracePeriod=30
Oct 30 21:19:35.453048 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:35.453015    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:35.950961 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:35.950917    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk" podUID="8ef92562-84d6-4d5e-b1c3-80d79f97100c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.170:8443/health\": dial tcp 10.132.0.170:8443: connect: connection refused"
Oct 30 21:19:36.523162 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:36.523133    5002 generic.go:356] "Generic (PLEG): container finished" podID="8ef92562-84d6-4d5e-b1c3-80d79f97100c" containerID="ea88c2818a0f64e0310867484be6e4f41d7945043e6339acf3d9b01bb001e42f" exitCode=0
Oct 30 21:19:36.523581 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:36.523211    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk" event={"ID":"8ef92562-84d6-4d5e-b1c3-80d79f97100c","Type":"ContainerDied","Data":"ea88c2818a0f64e0310867484be6e4f41d7945043e6339acf3d9b01bb001e42f"}
Oct 30 21:19:36.971568 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:36.971512    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:37.075780 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.075756    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:19:37.117146 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:19:37.117112    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:19:37.190073 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.189995    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef92562-84d6-4d5e-b1c3-80d79f97100c-kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/8ef92562-84d6-4d5e-b1c3-80d79f97100c-kube-api-access-cxb7j\") pod \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") "
Oct 30 21:19:37.190073 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.190040    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef92562-84d6-4d5e-b1c3-80d79f97100c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92562-84d6-4d5e-b1c3-80d79f97100c-openshift-service-ca\") pod \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") "
Oct 30 21:19:37.190269 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.190224    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-metrics-certs\") pod \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") "
Oct 30 21:19:37.190327 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.190303    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-maas-api-tls\") pod \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\" (UID: \"8ef92562-84d6-4d5e-b1c3-80d79f97100c\") "
Oct 30 21:19:37.190419 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.190400    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef92562-84d6-4d5e-b1c3-80d79f97100c-openshift-service-ca" pod "8ef92562-84d6-4d5e-b1c3-80d79f97100c" (UID: "8ef92562-84d6-4d5e-b1c3-80d79f97100c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:37.190700 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.190675    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92562-84d6-4d5e-b1c3-80d79f97100c-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.192299 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.192270    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-metrics-certs" pod "8ef92562-84d6-4d5e-b1c3-80d79f97100c" (UID: "8ef92562-84d6-4d5e-b1c3-80d79f97100c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:37.192436 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.192415    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-maas-api-tls" pod "8ef92562-84d6-4d5e-b1c3-80d79f97100c" (UID: "8ef92562-84d6-4d5e-b1c3-80d79f97100c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:37.192484 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.192436    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef92562-84d6-4d5e-b1c3-80d79f97100c-kube-api-access-cxb7j" pod "8ef92562-84d6-4d5e-b1c3-80d79f97100c" (UID: "8ef92562-84d6-4d5e-b1c3-80d79f97100c"). InnerVolumeSpecName "kube-api-access-cxb7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:37.249061 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.249028    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:37.291878 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.291849    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/8ef92562-84d6-4d5e-b1c3-80d79f97100c-kube-api-access-cxb7j\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.291878 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.291876    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.292146 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.291886    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef92562-84d6-4d5e-b1c3-80d79f97100c-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.530314 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.530282    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"
Oct 30 21:19:37.530314 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.530309    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk" event={"ID":"8ef92562-84d6-4d5e-b1c3-80d79f97100c","Type":"ContainerDied","Data":"a0b15a7c45be64bf7c9c37d15d4dc173795c622402151b7d2f8653d329290ac4"}
Oct 30 21:19:37.530874 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.530363    5002 scope.go:122] "RemoveContainer" containerID="ea88c2818a0f64e0310867484be6e4f41d7945043e6339acf3d9b01bb001e42f"
Oct 30 21:19:37.695831 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.695788    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"]
Oct 30 21:19:37.701045 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.701018    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-819d1b-789cb8758c-8zcdk"]
Oct 30 21:19:37.931002 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.930919    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" podUID="5cc47e4f-3504-4410-9265-31ae3ad03e32" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:19:37.931155 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.930930    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" podUID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:19:37.966521 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:37.966482    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef92562-84d6-4d5e-b1c3-80d79f97100c" path="/var/lib/kubelet/pods/8ef92562-84d6-4d5e-b1c3-80d79f97100c/volumes"
Oct 30 21:19:38.164088 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:38.164058    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:39.983178 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:39.983144    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:19:43.803024 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:43.802989    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:43.998462 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:43.998427    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:44.175374 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:44.175291    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:44.513877 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:44.513843    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:47.931169 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:47.931130    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" podUID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:19:47.931169 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:47.931156    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" podUID="5cc47e4f-3504-4410-9265-31ae3ad03e32" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:19:54.073890 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.073819    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:19:54.165842 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.165814    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:19:54.169977 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.169960    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694a3165-0de0-43f8-bf64-1f26d29ee8b9-kube-api-access-xkx2c\" (UniqueName: \"kubernetes.io/projected/694a3165-0de0-43f8-bf64-1f26d29ee8b9-kube-api-access-xkx2c\") pod \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") "
Oct 30 21:19:54.170064 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.169997    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694a3165-0de0-43f8-bf64-1f26d29ee8b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a3165-0de0-43f8-bf64-1f26d29ee8b9-plugins-config-volume\") pod \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\" (UID: \"694a3165-0de0-43f8-bf64-1f26d29ee8b9\") "
Oct 30 21:19:54.170362 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.170335    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694a3165-0de0-43f8-bf64-1f26d29ee8b9-plugins-config-volume" pod "694a3165-0de0-43f8-bf64-1f26d29ee8b9" (UID: "694a3165-0de0-43f8-bf64-1f26d29ee8b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:54.172130 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.172108    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694a3165-0de0-43f8-bf64-1f26d29ee8b9-kube-api-access-xkx2c" pod "694a3165-0de0-43f8-bf64-1f26d29ee8b9" (UID: "694a3165-0de0-43f8-bf64-1f26d29ee8b9"). InnerVolumeSpecName "kube-api-access-xkx2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:54.271191 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.271158    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc47e4f-3504-4410-9265-31ae3ad03e32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc47e4f-3504-4410-9265-31ae3ad03e32-plugins-config-volume\") pod \"5cc47e4f-3504-4410-9265-31ae3ad03e32\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") "
Oct 30 21:19:54.271366 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.271200    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc47e4f-3504-4410-9265-31ae3ad03e32-kube-api-access-kxm5b\" (UniqueName: \"kubernetes.io/projected/5cc47e4f-3504-4410-9265-31ae3ad03e32-kube-api-access-kxm5b\") pod \"5cc47e4f-3504-4410-9265-31ae3ad03e32\" (UID: \"5cc47e4f-3504-4410-9265-31ae3ad03e32\") "
Oct 30 21:19:54.271409 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.271394    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkx2c\" (UniqueName: \"kubernetes.io/projected/694a3165-0de0-43f8-bf64-1f26d29ee8b9-kube-api-access-xkx2c\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.271444 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.271414    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a3165-0de0-43f8-bf64-1f26d29ee8b9-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.271642 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.271617    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc47e4f-3504-4410-9265-31ae3ad03e32-plugins-config-volume" pod "5cc47e4f-3504-4410-9265-31ae3ad03e32" (UID: "5cc47e4f-3504-4410-9265-31ae3ad03e32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:54.273459 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.273441    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc47e4f-3504-4410-9265-31ae3ad03e32-kube-api-access-kxm5b" pod "5cc47e4f-3504-4410-9265-31ae3ad03e32" (UID: "5cc47e4f-3504-4410-9265-31ae3ad03e32"). InnerVolumeSpecName "kube-api-access-kxm5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:54.372872 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.372800    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc47e4f-3504-4410-9265-31ae3ad03e32-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.372872 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.372827    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxm5b\" (UniqueName: \"kubernetes.io/projected/5cc47e4f-3504-4410-9265-31ae3ad03e32-kube-api-access-kxm5b\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.635339 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.635245    5002 generic.go:356] "Generic (PLEG): container finished" podID="5cc47e4f-3504-4410-9265-31ae3ad03e32" containerID="125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b" exitCode=0
Oct 30 21:19:54.635339 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.635287    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"
Oct 30 21:19:54.635339 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.635326    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" event={"ID":"5cc47e4f-3504-4410-9265-31ae3ad03e32","Type":"ContainerDied","Data":"125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b"}
Oct 30 21:19:54.635671 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.635382    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg" event={"ID":"5cc47e4f-3504-4410-9265-31ae3ad03e32","Type":"ContainerDied","Data":"06b7904d87eb10246c4479531b578f2ce48529b114ce33826025c02ad1c40c35"}
Oct 30 21:19:54.635671 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.635410    5002 scope.go:122] "RemoveContainer" containerID="125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b"
Oct 30 21:19:54.636721 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.636700    5002 generic.go:356] "Generic (PLEG): container finished" podID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" containerID="459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b" exitCode=0
Oct 30 21:19:54.636832 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.636742    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" event={"ID":"694a3165-0de0-43f8-bf64-1f26d29ee8b9","Type":"ContainerDied","Data":"459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b"}
Oct 30 21:19:54.636832 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.636749    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"
Oct 30 21:19:54.636832 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.636779    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r" event={"ID":"694a3165-0de0-43f8-bf64-1f26d29ee8b9","Type":"ContainerDied","Data":"57e81e20975ddbba4d9af01e0ba8c9cac43e05cc0b485ba8ffa65937d2d4c1d6"}
Oct 30 21:19:54.646997 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.646982    5002 scope.go:122] "RemoveContainer" containerID="125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b"
Oct 30 21:19:54.647286 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:19:54.647269    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b\": container with ID starting with 125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b not found: ID does not exist" containerID="125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b"
Oct 30 21:19:54.647333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.647296    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b"} err="failed to get container status \"125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b\": rpc error: code = NotFound desc = could not find container \"125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b\": container with ID starting with 125c195f138d3b1e6821c80455c2c8941a5b2d98a966e296f7bb7f7410ec096b not found: ID does not exist"
Oct 30 21:19:54.647333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.647313    5002 scope.go:122] "RemoveContainer" containerID="459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b"
Oct 30 21:19:54.656308 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.656289    5002 scope.go:122] "RemoveContainer" containerID="459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b"
Oct 30 21:19:54.656592 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:19:54.656569    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b\": container with ID starting with 459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b not found: ID does not exist" containerID="459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b"
Oct 30 21:19:54.656712 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.656596    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b"} err="failed to get container status \"459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b\": rpc error: code = NotFound desc = could not find container \"459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b\": container with ID starting with 459a9be9d1cf187c121dc9c260189479aa01e97decd943aa229591cb07eb2f5b not found: ID does not exist"
Oct 30 21:19:54.801464 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.801425    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"]
Oct 30 21:19:54.851876 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.851849    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg"]
Oct 30 21:19:54.920584 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.920489    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"]
Oct 30 21:19:54.926479 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:54.926434    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r"]
Oct 30 21:19:55.965743 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:55.965706    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc47e4f-3504-4410-9265-31ae3ad03e32" path="/var/lib/kubelet/pods/5cc47e4f-3504-4410-9265-31ae3ad03e32/volumes"
Oct 30 21:19:55.966160 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:55.966145    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" path="/var/lib/kubelet/pods/694a3165-0de0-43f8-bf64-1f26d29ee8b9/volumes"
Oct 30 21:19:58.469213 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:58.469184    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:58.929758 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:58.929668    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:59.481477 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:19:59.481440    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:04.484825 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.484793    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"]
Oct 30 21:20:04.485309 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.485009    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh" podUID="4d66661c-3d8b-4af2-af19-00af4f63c714" containerName="manager" containerID="cri-o://12d192caec54458bb3b0356c9da40f32e12dcf0e3b7c829e814352d304d1928e" gracePeriod=10
Oct 30 21:20:04.676903 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.676862    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:04.697966 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.697935    5002 generic.go:356] "Generic (PLEG): container finished" podID="4d66661c-3d8b-4af2-af19-00af4f63c714" containerID="12d192caec54458bb3b0356c9da40f32e12dcf0e3b7c829e814352d304d1928e" exitCode=0
Oct 30 21:20:04.698134 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.698003    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh" event={"ID":"4d66661c-3d8b-4af2-af19-00af4f63c714","Type":"ContainerDied","Data":"12d192caec54458bb3b0356c9da40f32e12dcf0e3b7c829e814352d304d1928e"}
Oct 30 21:20:04.747018 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.746958    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 21:20:04.882937 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.882905    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d66661c-3d8b-4af2-af19-00af4f63c714-kube-api-access-8js5m\" (UniqueName: \"kubernetes.io/projected/4d66661c-3d8b-4af2-af19-00af4f63c714-kube-api-access-8js5m\") pod \"4d66661c-3d8b-4af2-af19-00af4f63c714\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") "
Oct 30 21:20:04.883234 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.883213    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d66661c-3d8b-4af2-af19-00af4f63c714-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d66661c-3d8b-4af2-af19-00af4f63c714-extensions-socket-volume\") pod \"4d66661c-3d8b-4af2-af19-00af4f63c714\" (UID: \"4d66661c-3d8b-4af2-af19-00af4f63c714\") "
Oct 30 21:20:04.883835 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.883808    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d66661c-3d8b-4af2-af19-00af4f63c714-extensions-socket-volume" pod "4d66661c-3d8b-4af2-af19-00af4f63c714" (UID: "4d66661c-3d8b-4af2-af19-00af4f63c714"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:04.885299 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.885264    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d66661c-3d8b-4af2-af19-00af4f63c714-kube-api-access-8js5m" pod "4d66661c-3d8b-4af2-af19-00af4f63c714" (UID: "4d66661c-3d8b-4af2-af19-00af4f63c714"). InnerVolumeSpecName "kube-api-access-8js5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:04.985001 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.984967    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8js5m\" (UniqueName: \"kubernetes.io/projected/4d66661c-3d8b-4af2-af19-00af4f63c714-kube-api-access-8js5m\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:04.985001 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:04.985001    5002 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d66661c-3d8b-4af2-af19-00af4f63c714-extensions-socket-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:05.708286 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:05.708250    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"
Oct 30 21:20:05.708794 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:05.708286    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh" event={"ID":"4d66661c-3d8b-4af2-af19-00af4f63c714","Type":"ContainerDied","Data":"1fce9fbc4dce795b0b748a9e157cfa786d9a63fe806bb1d620ce78ce8fc4c975"}
Oct 30 21:20:05.708794 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:05.708360    5002 scope.go:122] "RemoveContainer" containerID="12d192caec54458bb3b0356c9da40f32e12dcf0e3b7c829e814352d304d1928e"
Oct 30 21:20:05.926352 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:05.926321    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"]
Oct 30 21:20:05.975998 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:05.975966    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br5mh"]
Oct 30 21:20:07.115286 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:20:07.115243    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:20:07.965640 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:07.965604    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d66661c-3d8b-4af2-af19-00af4f63c714" path="/var/lib/kubelet/pods/4d66661c-3d8b-4af2-af19-00af4f63c714/volumes"
Oct 30 21:20:16.045094 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:16.045005    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"]
Oct 30 21:20:16.095201 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:16.095166    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sbg25"]
Oct 30 21:20:17.964945 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:17.964886    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="537924bf-9989-44a0-99ce-1529d8910055" path="/var/lib/kubelet/pods/537924bf-9989-44a0-99ce-1529d8910055/volumes"
Oct 30 21:20:37.115106 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:20:37.115068    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:20:44.910082 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:20:44.910034    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:07.115370 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:21:07.115331    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:21:08.191969 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.191942    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"]
Oct 30 21:21:08.192439 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192375    5002 cpu_manager.go:414] "containerMap: removing container" podUID="8ef92562-84d6-4d5e-b1c3-80d79f97100c" containerName="maas-api"
Oct 30 21:21:08.192439 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192391    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef92562-84d6-4d5e-b1c3-80d79f97100c" containerName="maas-api"
Oct 30 21:21:08.192439 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192410    5002 cpu_manager.go:414] "containerMap: removing container" podUID="5cc47e4f-3504-4410-9265-31ae3ad03e32" containerName="payload-pre-processing"
Oct 30 21:21:08.192439 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192419    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc47e4f-3504-4410-9265-31ae3ad03e32" containerName="payload-pre-processing"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192452    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4d66661c-3d8b-4af2-af19-00af4f63c714" containerName="manager"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192458    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d66661c-3d8b-4af2-af19-00af4f63c714" containerName="manager"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192469    5002 cpu_manager.go:414] "containerMap: removing container" podUID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" containerName="payload-processing"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192474    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" containerName="payload-processing"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192575    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef92562-84d6-4d5e-b1c3-80d79f97100c" containerName="maas-api"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192587    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc47e4f-3504-4410-9265-31ae3ad03e32" containerName="payload-pre-processing"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192596    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d66661c-3d8b-4af2-af19-00af4f63c714" containerName="manager"
Oct 30 21:21:08.192639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.192602    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="694a3165-0de0-43f8-bf64-1f26d29ee8b9" containerName="payload-processing"
Oct 30 21:21:08.195717 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.195700    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.247005 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.246974    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"]
Oct 30 21:21:08.294887 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.294854    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vhm96\""
Oct 30 21:21:08.301435 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.301397    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8943322-e55c-454a-a806-55ac57874dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8kf5n\" (UID: \"f8943322-e55c-454a-a806-55ac57874dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.301674 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.301651    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvc7\" (UniqueName: \"kubernetes.io/projected/f8943322-e55c-454a-a806-55ac57874dbd-kube-api-access-dsvc7\") pod \"kuadrant-operator-controller-manager-678d87766f-8kf5n\" (UID: \"f8943322-e55c-454a-a806-55ac57874dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.402376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.402337    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvc7\" (UniqueName: \"kubernetes.io/projected/f8943322-e55c-454a-a806-55ac57874dbd-kube-api-access-dsvc7\") pod \"kuadrant-operator-controller-manager-678d87766f-8kf5n\" (UID: \"f8943322-e55c-454a-a806-55ac57874dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.402580 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.402398    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8943322-e55c-454a-a806-55ac57874dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8kf5n\" (UID: \"f8943322-e55c-454a-a806-55ac57874dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.402858 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.402840    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8943322-e55c-454a-a806-55ac57874dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8kf5n\" (UID: \"f8943322-e55c-454a-a806-55ac57874dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.463919 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.463889    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvc7\" (UniqueName: \"kubernetes.io/projected/f8943322-e55c-454a-a806-55ac57874dbd-kube-api-access-dsvc7\") pod \"kuadrant-operator-controller-manager-678d87766f-8kf5n\" (UID: \"f8943322-e55c-454a-a806-55ac57874dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.509025 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.508992    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:08.982248 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:08.982216    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"]
Oct 30 21:21:09.020065 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:21:09.020039    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8943322_e55c_454a_a806_55ac57874dbd.slice/crio-c7bb169d1608bf74b0932dabb5a56c3b2b8bfe5c1a7a3f88a6d8c0dd8f82008c WatchSource:0}: Error finding container c7bb169d1608bf74b0932dabb5a56c3b2b8bfe5c1a7a3f88a6d8c0dd8f82008c: Status 404 returned error can't find the container with id c7bb169d1608bf74b0932dabb5a56c3b2b8bfe5c1a7a3f88a6d8c0dd8f82008c
Oct 30 21:21:09.067518 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:09.067493    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n" event={"ID":"f8943322-e55c-454a-a806-55ac57874dbd","Type":"ContainerStarted","Data":"c7bb169d1608bf74b0932dabb5a56c3b2b8bfe5c1a7a3f88a6d8c0dd8f82008c"}
Oct 30 21:21:10.074472 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:10.074428    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n" event={"ID":"f8943322-e55c-454a-a806-55ac57874dbd","Type":"ContainerStarted","Data":"6f66547c5de7e5eed84d435bd096a7524bff1ecb928d71ccaab18dc82e8b9685"}
Oct 30 21:21:10.074986 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:10.074522    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:10.299688 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:10.299630    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n" podStartSLOduration=2.299613477 podStartE2EDuration="2.299613477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:10.247002504 +0000 UTC m=+3696.396053514" watchObservedRunningTime="2026-10-30 21:21:10.299613477 +0000 UTC m=+3696.448664462"
Oct 30 21:21:21.083098 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:21.083064    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8kf5n"
Oct 30 21:21:28.236618 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:28.236585    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:21:35.313124 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:35.313090    5002 scope.go:122] "RemoveContainer" containerID="e2eb24e0107f474d39b16ad88aa36b26b89fab0ab054c3f969dde199d2519a2b"
Oct 30 21:21:37.115112 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:21:37.115076    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:21:45.236678 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:45.236593    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:45.634161 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:45.634073    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:21:55.171387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:21:55.171354    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:22:07.115162 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:22:07.115120    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:22:12.038954 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:12.038919    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:14.667375 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:14.667340    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:14.926064 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:14.925986    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:22:15.371672 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:15.371635    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:15.556807 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:15.556762    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:19.323086 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:19.323051    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:19.543996 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:19.543958    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:22.639220 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:22.639180    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:22.822477 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:22.822444    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:23.153666 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.153629    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"]
Oct 30 21:22:23.157718 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.157698    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.214482 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.214448    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-87bb59\""
Oct 30 21:22:23.264804 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.264763    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b148dbd6-dockercfg-lgw54\""
Oct 30 21:22:23.267205 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.267181    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"]
Oct 30 21:22:23.277284 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.277263    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.277342 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.277293    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dsm\" (UniqueName: \"kubernetes.io/projected/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-kube-api-access-l8dsm\") pod \"payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.311569 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.311504    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"]
Oct 30 21:22:23.315718 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.315697    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:23.371801 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.371769    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"]
Oct 30 21:22:23.378643 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.378612    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.378828 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.378658    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dsm\" (UniqueName: \"kubernetes.io/projected/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-kube-api-access-l8dsm\") pod \"payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.379389 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.379364    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.437360 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.437275    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dsm\" (UniqueName: \"kubernetes.io/projected/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-kube-api-access-l8dsm\") pod \"payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.470935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.470902    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:23.479624 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.479589    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda494a-2c35-43c1-b12a-b23f5024cfe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:23.479721 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.479700    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vt2k\" (UniqueName: \"kubernetes.io/projected/7dda494a-2c35-43c1-b12a-b23f5024cfe3-kube-api-access-5vt2k\") pod \"payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:23.581327 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.581289    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda494a-2c35-43c1-b12a-b23f5024cfe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:23.581504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.581357    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vt2k\" (UniqueName: \"kubernetes.io/projected/7dda494a-2c35-43c1-b12a-b23f5024cfe3-kube-api-access-5vt2k\") pod \"payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:23.582092 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.582067    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda494a-2c35-43c1-b12a-b23f5024cfe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:23.639783 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.639755    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vt2k\" (UniqueName: \"kubernetes.io/projected/7dda494a-2c35-43c1-b12a-b23f5024cfe3-kube-api-access-5vt2k\") pod \"payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:23.850934 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.850894    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"]
Oct 30 21:22:23.852311 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:22:23.852287    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc9b7e5_b361_4934_a80c_e1e3925e4a9a.slice/crio-d77f6114b38bd14c5839ce3ee4dbbc31c26f98017099e7c87b5a40d248126487 WatchSource:0}: Error finding container d77f6114b38bd14c5839ce3ee4dbbc31c26f98017099e7c87b5a40d248126487: Status 404 returned error can't find the container with id d77f6114b38bd14c5839ce3ee4dbbc31c26f98017099e7c87b5a40d248126487
Oct 30 21:22:23.929142 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:23.929111    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:24.309039 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.309011    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"]
Oct 30 21:22:24.310296 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:22:24.310268    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dda494a_2c35_43c1_b12a_b23f5024cfe3.slice/crio-85432084da71a3194466cbc761f0c2a6e533ef03de7c99178e0317dc93ae90ae WatchSource:0}: Error finding container 85432084da71a3194466cbc761f0c2a6e533ef03de7c99178e0317dc93ae90ae: Status 404 returned error can't find the container with id 85432084da71a3194466cbc761f0c2a6e533ef03de7c99178e0317dc93ae90ae
Oct 30 21:22:24.539949 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.539914    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" event={"ID":"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a","Type":"ContainerStarted","Data":"4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637"}
Oct 30 21:22:24.540151 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.539965    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" event={"ID":"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a","Type":"ContainerStarted","Data":"d77f6114b38bd14c5839ce3ee4dbbc31c26f98017099e7c87b5a40d248126487"}
Oct 30 21:22:24.540151 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.540103    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:24.541590 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.541565    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" event={"ID":"7dda494a-2c35-43c1-b12a-b23f5024cfe3","Type":"ContainerStarted","Data":"8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4"}
Oct 30 21:22:24.541685 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.541611    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:24.541685 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.541632    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" event={"ID":"7dda494a-2c35-43c1-b12a-b23f5024cfe3","Type":"ContainerStarted","Data":"85432084da71a3194466cbc761f0c2a6e533ef03de7c99178e0317dc93ae90ae"}
Oct 30 21:22:24.656123 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:24.656014    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" podStartSLOduration=1.65600062 podStartE2EDuration="1.65600062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:24.655685506 +0000 UTC m=+3770.804736494" watchObservedRunningTime="2026-10-30 21:22:24.65600062 +0000 UTC m=+3770.805051607"
Oct 30 21:22:25.431317 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.431266    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" podStartSLOduration=2.431252241 podStartE2EDuration="2.431252241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:24.780404081 +0000 UTC m=+3770.929455070" watchObservedRunningTime="2026-10-30 21:22:25.431252241 +0000 UTC m=+3771.580303229"
Oct 30 21:22:25.431519 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.431485    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"]
Oct 30 21:22:25.435240 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.435222    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.489911 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.489877    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:25.490104 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.489934    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:25.498184 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.498150    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.498326 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.498268    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfv5c\" (UniqueName: \"kubernetes.io/projected/f6dd0952-e4d8-4262-b146-2734e921094b-kube-api-access-gfv5c\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.498326 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.498315    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0952-e4d8-4262-b146-2734e921094b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.498438 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.498340    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.540228 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.540188    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-87bb59-serving-cert\""
Oct 30 21:22:25.540415 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.540272    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xgxz5\""
Oct 30 21:22:25.540493 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.540421    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:22:25.543501 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.543458    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"]
Oct 30 21:22:25.598975 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.598937    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfv5c\" (UniqueName: \"kubernetes.io/projected/f6dd0952-e4d8-4262-b146-2734e921094b-kube-api-access-gfv5c\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.599142 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.598981    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0952-e4d8-4262-b146-2734e921094b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.599142 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.599049    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.599285 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:22:25.599181    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-serving-cert: secret "maas-api-e2e-worker-single-main-87bb59-serving-cert" not found
Oct 30 21:22:25.599285 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:22:25.599259    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls podName:f6dd0952-e4d8-4262-b146-2734e921094b nodeName:}" failed. No retries permitted until 2026-10-30 21:22:26.099234832 +0000 UTC m=+3772.248285808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls") pod "maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq" (UID: "f6dd0952-e4d8-4262-b146-2734e921094b") : secret "maas-api-e2e-worker-single-main-87bb59-serving-cert" not found
Oct 30 21:22:25.599404 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.599379    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.600056 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.600034    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0952-e4d8-4262-b146-2734e921094b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.602017 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.601993    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.662674 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.657377    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfv5c\" (UniqueName: \"kubernetes.io/projected/f6dd0952-e4d8-4262-b146-2734e921094b-kube-api-access-gfv5c\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:25.685987 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.682842    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:25.855763 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:25.855724    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:26.037799 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:26.037761    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:26.104885 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:26.104853    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:26.107397 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:26.107370    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:26.351934 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:26.351854    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:26.728356 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:26.728330    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"]
Oct 30 21:22:26.729056 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:22:26.729029    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dd0952_e4d8_4262_b146_2734e921094b.slice/crio-eabc29e876ee7dee4324c6807b3813942e01c5dc453f1016fb6b94d802eee8a6 WatchSource:0}: Error finding container eabc29e876ee7dee4324c6807b3813942e01c5dc453f1016fb6b94d802eee8a6: Status 404 returned error can't find the container with id eabc29e876ee7dee4324c6807b3813942e01c5dc453f1016fb6b94d802eee8a6
Oct 30 21:22:27.561311 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:27.561268    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq" event={"ID":"f6dd0952-e4d8-4262-b146-2734e921094b","Type":"ContainerStarted","Data":"eabc29e876ee7dee4324c6807b3813942e01c5dc453f1016fb6b94d802eee8a6"}
Oct 30 21:22:28.568588 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:28.568549    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq" event={"ID":"f6dd0952-e4d8-4262-b146-2734e921094b","Type":"ContainerStarted","Data":"358f36916613a8dbf08189f18476ad1358bfc7847ca7d009f293cfd8dab824da"}
Oct 30 21:22:28.569057 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:28.568736    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:28.687611 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:28.687556    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq" podStartSLOduration=2.799959755 podStartE2EDuration="3.687542317s" totalImagesPullingTime="887.582562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:28.687307058 +0000 UTC m=+3774.836358046" watchObservedRunningTime="2026-10-30 21:22:28.687542317 +0000 UTC m=+3774.836593296"
Oct 30 21:22:30.626660 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:30.626614    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:33.527081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:33.527041    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:22:34.579880 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:34.579848    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:22:35.550759 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:35.550728    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:35.552600 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:35.552571    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:22:35.552929 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:35.552909    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:22:37.115621 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:22:37.115581    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:22:38.858555 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:38.858498    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:40.463527 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:40.463485    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:44.307236 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:44.307167    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:44.994685 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:44.994650    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:45.219034 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:45.218999    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:45.556154 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:45.556119    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:45.762055 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:45.762016    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:47.532333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:47.532299    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:50.548268 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:50.548183    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:51.985821 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:51.985698    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:22:55.411969 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:22:55.411927    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:00.233137 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:00.233094    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:04.779118 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:04.779083    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:04.993031 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:04.992991    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:05.172503 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:05.172379    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:07.115778 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:23:07.115744    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:23:09.864069 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:09.864033    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:14.710014 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:14.709933    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:19.516181 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:19.516140    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:20.219350 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:20.219299    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:23:24.338376 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:24.338341    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:29.139453 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:29.139416    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:33.988879 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:33.988844    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:37.139122 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:23:37.139085    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:23:38.732823 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:38.732764    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:43.649513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:43.649479    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:45.423677 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:45.423644    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:45.874695 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:45.874663    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:47.280627 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:47.280591    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:23:48.929408 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:48.929368    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:50.460832 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:50.460796    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:53.968797 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:53.968762    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:57.460312 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:57.460276    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"]
Oct 30 21:23:57.567026 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:57.566996    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"]
Oct 30 21:23:57.871767 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:57.871693    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:58.045035 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:58.045001    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:58.821840 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:23:58.821807    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:02.460830 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:02.460787    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" podUID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" containerName="payload-processing" containerID="cri-o://4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637" gracePeriod=25
Oct 30 21:24:02.567605 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:02.567571    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" podUID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" containerName="payload-pre-processing" containerID="cri-o://8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4" gracePeriod=25
Oct 30 21:24:02.924474 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:02.924397    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"]
Oct 30 21:24:02.924639 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:02.924606    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq" podUID="f6dd0952-e4d8-4262-b146-2734e921094b" containerName="maas-api" containerID="cri-o://358f36916613a8dbf08189f18476ad1358bfc7847ca7d009f293cfd8dab824da" gracePeriod=30
Oct 30 21:24:03.069011 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:03.068978    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:04.112019 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.111990    5002 generic.go:356] "Generic (PLEG): container finished" podID="f6dd0952-e4d8-4262-b146-2734e921094b" containerID="358f36916613a8dbf08189f18476ad1358bfc7847ca7d009f293cfd8dab824da" exitCode=0
Oct 30 21:24:04.112468 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.112052    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq" event={"ID":"f6dd0952-e4d8-4262-b146-2734e921094b","Type":"ContainerDied","Data":"358f36916613a8dbf08189f18476ad1358bfc7847ca7d009f293cfd8dab824da"}
Oct 30 21:24:04.180143 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.180110    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:24:04.284379 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.283377    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dd0952-e4d8-4262-b146-2734e921094b-kube-api-access-gfv5c\" (UniqueName: \"kubernetes.io/projected/f6dd0952-e4d8-4262-b146-2734e921094b-kube-api-access-gfv5c\") pod \"f6dd0952-e4d8-4262-b146-2734e921094b\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") "
Oct 30 21:24:04.284379 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.283488    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-metrics-certs\") pod \"f6dd0952-e4d8-4262-b146-2734e921094b\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") "
Oct 30 21:24:04.284379 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.283559    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls\") pod \"f6dd0952-e4d8-4262-b146-2734e921094b\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") "
Oct 30 21:24:04.284379 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.283658    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dd0952-e4d8-4262-b146-2734e921094b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0952-e4d8-4262-b146-2734e921094b-openshift-service-ca\") pod \"f6dd0952-e4d8-4262-b146-2734e921094b\" (UID: \"f6dd0952-e4d8-4262-b146-2734e921094b\") "
Oct 30 21:24:04.285172 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.284844    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dd0952-e4d8-4262-b146-2734e921094b-openshift-service-ca" pod "f6dd0952-e4d8-4262-b146-2734e921094b" (UID: "f6dd0952-e4d8-4262-b146-2734e921094b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:04.288361 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.288334    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-metrics-certs" pod "f6dd0952-e4d8-4262-b146-2734e921094b" (UID: "f6dd0952-e4d8-4262-b146-2734e921094b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:04.291175 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.291144    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls" pod "f6dd0952-e4d8-4262-b146-2734e921094b" (UID: "f6dd0952-e4d8-4262-b146-2734e921094b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:04.292986 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.292961    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dd0952-e4d8-4262-b146-2734e921094b-kube-api-access-gfv5c" pod "f6dd0952-e4d8-4262-b146-2734e921094b" (UID: "f6dd0952-e4d8-4262-b146-2734e921094b"). InnerVolumeSpecName "kube-api-access-gfv5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:04.385554 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.385506    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0952-e4d8-4262-b146-2734e921094b-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:04.385554 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.385551    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfv5c\" (UniqueName: \"kubernetes.io/projected/f6dd0952-e4d8-4262-b146-2734e921094b-kube-api-access-gfv5c\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:04.385696 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.385564    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:04.385696 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.385573    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd0952-e4d8-4262-b146-2734e921094b-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:04.594161 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.594089    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:04.869126 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:04.869052    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:05.118229 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.118200    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"
Oct 30 21:24:05.118229 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.118216    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq" event={"ID":"f6dd0952-e4d8-4262-b146-2734e921094b","Type":"ContainerDied","Data":"eabc29e876ee7dee4324c6807b3813942e01c5dc453f1016fb6b94d802eee8a6"}
Oct 30 21:24:05.118759 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.118274    5002 scope.go:122] "RemoveContainer" containerID="358f36916613a8dbf08189f18476ad1358bfc7847ca7d009f293cfd8dab824da"
Oct 30 21:24:05.283072 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.283048    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"]
Oct 30 21:24:05.333081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.333055    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-87bb59-5f9595ff6f-xw9kq"]
Oct 30 21:24:05.550571 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.550508    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" podUID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:05.550714 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.550526    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" podUID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:05.654093 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.654060    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:05.965448 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:05.965415    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dd0952-e4d8-4262-b146-2734e921094b" path="/var/lib/kubelet/pods/f6dd0952-e4d8-4262-b146-2734e921094b/volumes"
Oct 30 21:24:07.115897 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:24:07.115865    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:24:07.554131 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:07.554102    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:24:11.349337 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:11.349295    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:11.537852 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:11.537815    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:11.716055 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:11.716025    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:15.551121 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:15.551084    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" podUID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:15.551622 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:15.551160    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" podUID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:19.005558 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:19.005491    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:22.734187 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.734112    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:24:22.753123 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.753097    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-kube-api-access-l8dsm\" (UniqueName: \"kubernetes.io/projected/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-kube-api-access-l8dsm\") pod \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") "
Oct 30 21:24:22.753293 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.753279    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-plugins-config-volume\") pod \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\" (UID: \"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a\") "
Oct 30 21:24:22.753956 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.753915    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-plugins-config-volume" pod "fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" (UID: "fcc9b7e5-b361-4934-a80c-e1e3925e4a9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:22.756592 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.756562    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-kube-api-access-l8dsm" pod "fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" (UID: "fcc9b7e5-b361-4934-a80c-e1e3925e4a9a"). InnerVolumeSpecName "kube-api-access-l8dsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:22.820511 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.820483    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:24:22.855253 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.854585    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dda494a-2c35-43c1-b12a-b23f5024cfe3-kube-api-access-5vt2k\" (UniqueName: \"kubernetes.io/projected/7dda494a-2c35-43c1-b12a-b23f5024cfe3-kube-api-access-5vt2k\") pod \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") "
Oct 30 21:24:22.855253 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.854689    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dda494a-2c35-43c1-b12a-b23f5024cfe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda494a-2c35-43c1-b12a-b23f5024cfe3-plugins-config-volume\") pod \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\" (UID: \"7dda494a-2c35-43c1-b12a-b23f5024cfe3\") "
Oct 30 21:24:22.855253 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.855015    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:22.855253 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.855038    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dsm\" (UniqueName: \"kubernetes.io/projected/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a-kube-api-access-l8dsm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:22.855791 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.855742    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dda494a-2c35-43c1-b12a-b23f5024cfe3-plugins-config-volume" pod "7dda494a-2c35-43c1-b12a-b23f5024cfe3" (UID: "7dda494a-2c35-43c1-b12a-b23f5024cfe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:22.857560 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.857515    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dda494a-2c35-43c1-b12a-b23f5024cfe3-kube-api-access-5vt2k" pod "7dda494a-2c35-43c1-b12a-b23f5024cfe3" (UID: "7dda494a-2c35-43c1-b12a-b23f5024cfe3"). InnerVolumeSpecName "kube-api-access-5vt2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:22.956155 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.956126    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vt2k\" (UniqueName: \"kubernetes.io/projected/7dda494a-2c35-43c1-b12a-b23f5024cfe3-kube-api-access-5vt2k\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:22.956155 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:22.956158    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda494a-2c35-43c1-b12a-b23f5024cfe3-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:23.222097 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.222057    5002 generic.go:356] "Generic (PLEG): container finished" podID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" containerID="4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637" exitCode=0
Oct 30 21:24:23.222300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.222098    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"
Oct 30 21:24:23.222300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.222142    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" event={"ID":"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a","Type":"ContainerDied","Data":"4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637"}
Oct 30 21:24:23.222300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.222230    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj" event={"ID":"fcc9b7e5-b361-4934-a80c-e1e3925e4a9a","Type":"ContainerDied","Data":"d77f6114b38bd14c5839ce3ee4dbbc31c26f98017099e7c87b5a40d248126487"}
Oct 30 21:24:23.222300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.222261    5002 scope.go:122] "RemoveContainer" containerID="4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637"
Oct 30 21:24:23.223638 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.223611    5002 generic.go:356] "Generic (PLEG): container finished" podID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" containerID="8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4" exitCode=0
Oct 30 21:24:23.223760 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.223655    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"
Oct 30 21:24:23.223760 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.223664    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" event={"ID":"7dda494a-2c35-43c1-b12a-b23f5024cfe3","Type":"ContainerDied","Data":"8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4"}
Oct 30 21:24:23.223760 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.223703    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh" event={"ID":"7dda494a-2c35-43c1-b12a-b23f5024cfe3","Type":"ContainerDied","Data":"85432084da71a3194466cbc761f0c2a6e533ef03de7c99178e0317dc93ae90ae"}
Oct 30 21:24:23.232410 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.232389    5002 scope.go:122] "RemoveContainer" containerID="4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637"
Oct 30 21:24:23.232790 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:24:23.232750    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637\": container with ID starting with 4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637 not found: ID does not exist" containerID="4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637"
Oct 30 21:24:23.232887 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.232805    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637"} err="failed to get container status \"4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637\": rpc error: code = NotFound desc = could not find container \"4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637\": container with ID starting with 4a662109321743270118836a49f4ebf32b6d5bad30874ba5c887c3295f0ad637 not found: ID does not exist"
Oct 30 21:24:23.232887 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.232832    5002 scope.go:122] "RemoveContainer" containerID="8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4"
Oct 30 21:24:23.241473 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.241456    5002 scope.go:122] "RemoveContainer" containerID="8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4"
Oct 30 21:24:23.241735 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:24:23.241717    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4\": container with ID starting with 8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4 not found: ID does not exist" containerID="8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4"
Oct 30 21:24:23.241782 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.241741    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4"} err="failed to get container status \"8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4\": rpc error: code = NotFound desc = could not find container \"8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4\": container with ID starting with 8b0605a976ddef91b8e287564088d131dfba8a05c149a38be28bcc645e6616b4 not found: ID does not exist"
Oct 30 21:24:23.390480 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.390434    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"]
Oct 30 21:24:23.400548 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.400500    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj"]
Oct 30 21:24:23.437834 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.437805    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:24:23.505693 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.505659    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"]
Oct 30 21:24:23.516440 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.516404    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh"]
Oct 30 21:24:23.969165 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.969126    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" path="/var/lib/kubelet/pods/7dda494a-2c35-43c1-b12a-b23f5024cfe3/volumes"
Oct 30 21:24:23.969784 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:23.969656    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" path="/var/lib/kubelet/pods/fcc9b7e5-b361-4934-a80c-e1e3925e4a9a/volumes"
Oct 30 21:24:26.661708 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:26.661674    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:27.612510 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:27.612472    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:34.569232 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:34.569206    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:24:34.590964 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:34.590936    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:24:37.115447 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:24:37.115407    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:24:41.899276 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:41.899241    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:42.854452 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:42.854374    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:43.044424 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:43.044394    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:46.213407 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:24:46.213372    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:01.452647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.452613    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"]
Oct 30 21:25:01.453074 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453036    5002 cpu_manager.go:414] "containerMap: removing container" podUID="f6dd0952-e4d8-4262-b146-2734e921094b" containerName="maas-api"
Oct 30 21:25:01.453074 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453049    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dd0952-e4d8-4262-b146-2734e921094b" containerName="maas-api"
Oct 30 21:25:01.453153 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453080    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" containerName="payload-processing"
Oct 30 21:25:01.453153 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453085    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" containerName="payload-processing"
Oct 30 21:25:01.453153 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453094    5002 cpu_manager.go:414] "containerMap: removing container" podUID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" containerName="payload-pre-processing"
Oct 30 21:25:01.453153 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453099    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" containerName="payload-pre-processing"
Oct 30 21:25:01.453285 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453165    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc9b7e5-b361-4934-a80c-e1e3925e4a9a" containerName="payload-processing"
Oct 30 21:25:01.453285 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453173    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dda494a-2c35-43c1-b12a-b23f5024cfe3" containerName="payload-pre-processing"
Oct 30 21:25:01.453285 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.453182    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dd0952-e4d8-4262-b146-2734e921094b" containerName="maas-api"
Oct 30 21:25:01.456320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.456303    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.562561 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.562506    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-569edc7e-kserve-self-signed-certs\""
Oct 30 21:25:01.564967 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.564943    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"]
Oct 30 21:25:01.579455 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.579429    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:01.606851 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.606820    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-tmp-dir\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.607049 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.606889    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-home\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.607049 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.606980    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgmn\" (UniqueName: \"kubernetes.io/projected/41911488-b939-4a19-8c4c-d0ebb1232ef5-kube-api-access-crgmn\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.607049 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.607012    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-dshm\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.607179 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.607096    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-model-cache\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.607179 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.607156    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41911488-b939-4a19-8c4c-d0ebb1232ef5-tls-certs\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.707737 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.707665    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41911488-b939-4a19-8c4c-d0ebb1232ef5-tls-certs\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.707877 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.707747    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-tmp-dir\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.707877 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.707811    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-home\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.707877 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.707842    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgmn\" (UniqueName: \"kubernetes.io/projected/41911488-b939-4a19-8c4c-d0ebb1232ef5-kube-api-access-crgmn\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.708002 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.707888    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-dshm\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.708002 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.707932    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-model-cache\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.708240 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.708207    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-tmp-dir\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.708368 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.708258    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-home\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.708368 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.708285    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-model-cache\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.710266 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.710242    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-dshm\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.710410 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.710301    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41911488-b939-4a19-8c4c-d0ebb1232ef5-tls-certs\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.765020 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.764993    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgmn\" (UniqueName: \"kubernetes.io/projected/41911488-b939-4a19-8c4c-d0ebb1232ef5-kube-api-access-crgmn\") pod \"e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") " pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:01.768856 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:01.768835    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:02.163769 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:02.163746    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"]
Oct 30 21:25:02.165373 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:25:02.165345    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41911488_b939_4a19_8c4c_d0ebb1232ef5.slice/crio-f83a2bcd2f2322401fa0ec6e86f009860df508ceeac67aca99eebf7a21ea445e WatchSource:0}: Error finding container f83a2bcd2f2322401fa0ec6e86f009860df508ceeac67aca99eebf7a21ea445e: Status 404 returned error can't find the container with id f83a2bcd2f2322401fa0ec6e86f009860df508ceeac67aca99eebf7a21ea445e
Oct 30 21:25:02.489168 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:02.489128    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf" event={"ID":"41911488-b939-4a19-8c4c-d0ebb1232ef5","Type":"ContainerStarted","Data":"10edbd07d1003132d0f759889370b2fbc54dbb42409b87aa361f58298e6b69a7"}
Oct 30 21:25:02.489589 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:02.489180    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf" event={"ID":"41911488-b939-4a19-8c4c-d0ebb1232ef5","Type":"ContainerStarted","Data":"f83a2bcd2f2322401fa0ec6e86f009860df508ceeac67aca99eebf7a21ea445e"}
Oct 30 21:25:02.608194 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:02.608140    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf" podStartSLOduration=1.608125588 podStartE2EDuration="1.608125588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:02.605870223 +0000 UTC m=+3928.754921210" watchObservedRunningTime="2026-10-30 21:25:02.608125588 +0000 UTC m=+3928.757176576"
Oct 30 21:25:02.769526 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:02.769439    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:06.052906 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:06.052872    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:25:07.122782 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:25:07.122740    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:25:11.769271 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:11.769238    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:11.781573 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:11.781550    5002 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:12.610252 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:12.610217    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:25:28.656470 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:28.656433    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:28.835990 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:28.835953    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:36.649572 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:36.648002    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:25:37.122005 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:25:37.121963    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:25:46.599404 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:46.599371    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:48.307166 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:48.307136    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:48.694427 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:48.694333    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:52.419439 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:52.419402    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:25:55.414568 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:55.406423    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:55.415188 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:25:55.414791    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"]
Oct 30 21:26:07.114765 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:26:07.114719    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:26:07.806395 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:07.806357    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:26:10.926958 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:10.926923    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf" podUID="41911488-b939-4a19-8c4c-d0ebb1232ef5" containerName="main" containerID="cri-o://10edbd07d1003132d0f759889370b2fbc54dbb42409b87aa361f58298e6b69a7" gracePeriod=45
Oct 30 21:26:11.142385 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.142351    5002 generic.go:356] "Generic (PLEG): container finished" podID="41911488-b939-4a19-8c4c-d0ebb1232ef5" containerID="10edbd07d1003132d0f759889370b2fbc54dbb42409b87aa361f58298e6b69a7" exitCode=0
Oct 30 21:26:11.142551 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.142416    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf" event={"ID":"41911488-b939-4a19-8c4c-d0ebb1232ef5","Type":"ContainerDied","Data":"10edbd07d1003132d0f759889370b2fbc54dbb42409b87aa361f58298e6b69a7"}
Oct 30 21:26:11.182109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.182039    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:26:11.276268 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276232    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41911488-b939-4a19-8c4c-d0ebb1232ef5-tls-certs\" (UniqueName: \"kubernetes.io/secret/41911488-b939-4a19-8c4c-d0ebb1232ef5-tls-certs\") pod \"41911488-b939-4a19-8c4c-d0ebb1232ef5\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") "
Oct 30 21:26:11.276434 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276283    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-model-cache\") pod \"41911488-b939-4a19-8c4c-d0ebb1232ef5\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") "
Oct 30 21:26:11.276434 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276350    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-home\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-home\") pod \"41911488-b939-4a19-8c4c-d0ebb1232ef5\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") "
Oct 30 21:26:11.276434 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276388    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-dshm\") pod \"41911488-b939-4a19-8c4c-d0ebb1232ef5\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") "
Oct 30 21:26:11.276667 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276464    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41911488-b939-4a19-8c4c-d0ebb1232ef5-kube-api-access-crgmn\" (UniqueName: \"kubernetes.io/projected/41911488-b939-4a19-8c4c-d0ebb1232ef5-kube-api-access-crgmn\") pod \"41911488-b939-4a19-8c4c-d0ebb1232ef5\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") "
Oct 30 21:26:11.276667 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276503    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-tmp-dir\") pod \"41911488-b939-4a19-8c4c-d0ebb1232ef5\" (UID: \"41911488-b939-4a19-8c4c-d0ebb1232ef5\") "
Oct 30 21:26:11.276667 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276624    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-model-cache" pod "41911488-b939-4a19-8c4c-d0ebb1232ef5" (UID: "41911488-b939-4a19-8c4c-d0ebb1232ef5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.276836 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276673    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-home" pod "41911488-b939-4a19-8c4c-d0ebb1232ef5" (UID: "41911488-b939-4a19-8c4c-d0ebb1232ef5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.276931 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276909    5002 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-model-cache\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.277000 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276939    5002 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-home\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.277000 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.276943    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-tmp-dir" pod "41911488-b939-4a19-8c4c-d0ebb1232ef5" (UID: "41911488-b939-4a19-8c4c-d0ebb1232ef5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.278898 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.278863    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41911488-b939-4a19-8c4c-d0ebb1232ef5-kube-api-access-crgmn" pod "41911488-b939-4a19-8c4c-d0ebb1232ef5" (UID: "41911488-b939-4a19-8c4c-d0ebb1232ef5"). InnerVolumeSpecName "kube-api-access-crgmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:11.279115 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.279097    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-dshm" pod "41911488-b939-4a19-8c4c-d0ebb1232ef5" (UID: "41911488-b939-4a19-8c4c-d0ebb1232ef5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.279165 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.279105    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41911488-b939-4a19-8c4c-d0ebb1232ef5-tls-certs" pod "41911488-b939-4a19-8c4c-d0ebb1232ef5" (UID: "41911488-b939-4a19-8c4c-d0ebb1232ef5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:11.378247 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.378217    5002 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-dshm\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.378247 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.378245    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgmn\" (UniqueName: \"kubernetes.io/projected/41911488-b939-4a19-8c4c-d0ebb1232ef5-kube-api-access-crgmn\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.378432 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.378255    5002 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41911488-b939-4a19-8c4c-d0ebb1232ef5-tmp-dir\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.378432 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:11.378264    5002 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41911488-b939-4a19-8c4c-d0ebb1232ef5-tls-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:12.149171 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:12.149141    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf" event={"ID":"41911488-b939-4a19-8c4c-d0ebb1232ef5","Type":"ContainerDied","Data":"f83a2bcd2f2322401fa0ec6e86f009860df508ceeac67aca99eebf7a21ea445e"}
Oct 30 21:26:12.149171 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:12.149167    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"
Oct 30 21:26:12.149633 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:12.149182    5002 scope.go:122] "RemoveContainer" containerID="10edbd07d1003132d0f759889370b2fbc54dbb42409b87aa361f58298e6b69a7"
Oct 30 21:26:12.315028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:12.314953    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"]
Oct 30 21:26:12.365329 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:12.365264    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-569edc7e-kserve-7494cd6549-l82pf"]
Oct 30 21:26:13.965065 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:13.965026    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41911488-b939-4a19-8c4c-d0ebb1232ef5" path="/var/lib/kubelet/pods/41911488-b939-4a19-8c4c-d0ebb1232ef5/volumes"
Oct 30 21:26:19.318552 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:19.318502    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:26:21.806986 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:21.806914    5002 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mr976 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:21.807655 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:21.807619    5002 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976" podUID="037d4d37-70b9-4335-9b96-4e5c740a48bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.117324 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:26:37.117281    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:26:42.440081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:42.440044    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:26:43.978003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:43.977963    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"]
Oct 30 21:26:43.978387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:43.978358    5002 cpu_manager.go:414] "containerMap: removing container" podUID="41911488-b939-4a19-8c4c-d0ebb1232ef5" containerName="main"
Oct 30 21:26:43.978387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:43.978369    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41911488-b939-4a19-8c4c-d0ebb1232ef5" containerName="main"
Oct 30 21:26:43.978461 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:43.978454    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="41911488-b939-4a19-8c4c-d0ebb1232ef5" containerName="main"
Oct 30 21:26:43.981543 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:43.981515    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.002509 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.002485    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"]
Oct 30 21:26:44.005953 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.005938    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.039235 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.039212    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"]
Oct 30 21:26:44.050260 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.050240    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xgxz5\""
Oct 30 21:26:44.085801 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.085779    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz6w\" (UniqueName: \"kubernetes.io/projected/a2a4ac24-81a3-450c-bb62-beefd608d486-kube-api-access-gsz6w\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.085908 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.085843    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94038fcc-f568-4af3-adca-9528d7d65ce5-metrics-certs\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.085951 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.085917    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-maas-api-tls\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.085987 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.085957    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4ac24-81a3-450c-bb62-beefd608d486-openshift-service-ca\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.086023 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.086016    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94038fcc-f568-4af3-adca-9528d7d65ce5-maas-api-tls\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.086062 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.086036    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94038fcc-f568-4af3-adca-9528d7d65ce5-openshift-service-ca\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.086126 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.086110    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsn6q\" (UniqueName: \"kubernetes.io/projected/94038fcc-f568-4af3-adca-9528d7d65ce5-kube-api-access-tsn6q\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.086163 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.086151    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-metrics-certs\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.087132 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.087110    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:26:44.087233 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.087147    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:26:44.087233 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.087161    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:44.087315 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.087240    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:44.089491 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.089472    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"]
Oct 30 21:26:44.187163 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187138    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz6w\" (UniqueName: \"kubernetes.io/projected/a2a4ac24-81a3-450c-bb62-beefd608d486-kube-api-access-gsz6w\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.187273 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187190    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94038fcc-f568-4af3-adca-9528d7d65ce5-metrics-certs\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.187273 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187207    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-maas-api-tls\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.187384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187307    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4ac24-81a3-450c-bb62-beefd608d486-openshift-service-ca\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.187384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187368    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94038fcc-f568-4af3-adca-9528d7d65ce5-maas-api-tls\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.187493 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187392    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94038fcc-f568-4af3-adca-9528d7d65ce5-openshift-service-ca\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.187493 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187430    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsn6q\" (UniqueName: \"kubernetes.io/projected/94038fcc-f568-4af3-adca-9528d7d65ce5-kube-api-access-tsn6q\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.187493 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.187472    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-metrics-certs\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.188211 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.188118    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94038fcc-f568-4af3-adca-9528d7d65ce5-openshift-service-ca\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.188660 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.188632    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4ac24-81a3-450c-bb62-beefd608d486-openshift-service-ca\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.190035 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.190013    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94038fcc-f568-4af3-adca-9528d7d65ce5-maas-api-tls\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.190329 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.190306    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94038fcc-f568-4af3-adca-9528d7d65ce5-metrics-certs\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.190385 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.190320    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-maas-api-tls\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.190655 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.190639    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-metrics-certs\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.245023 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.244976    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz6w\" (UniqueName: \"kubernetes.io/projected/a2a4ac24-81a3-450c-bb62-beefd608d486-kube-api-access-gsz6w\") pod \"maas-api-94dffd7db-cws4q\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.245104 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.245086    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsn6q\" (UniqueName: \"kubernetes.io/projected/94038fcc-f568-4af3-adca-9528d7d65ce5-kube-api-access-tsn6q\") pod \"maas-api-d979b6f8b-4bnbl\" (UID: \"94038fcc-f568-4af3-adca-9528d7d65ce5\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.302305 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.302280    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:44.318927 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.318892    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:44.720412 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.720388    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"]
Oct 30 21:26:44.721234 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:26:44.721206    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a4ac24_81a3_450c_bb62_beefd608d486.slice/crio-f394ab23771267d32685753165bb095c04e3f3aab35b8b2f349f78872aa54331 WatchSource:0}: Error finding container f394ab23771267d32685753165bb095c04e3f3aab35b8b2f349f78872aa54331: Status 404 returned error can't find the container with id f394ab23771267d32685753165bb095c04e3f3aab35b8b2f349f78872aa54331
Oct 30 21:26:44.722633 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.722614    5002 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:26:44.737333 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:44.737310    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"]
Oct 30 21:26:44.757661 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:26:44.757637    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94038fcc_f568_4af3_adca_9528d7d65ce5.slice/crio-61f079e328f526489be7a844121cc631eeebd9dba339a598944d622262ce0e65 WatchSource:0}: Error finding container 61f079e328f526489be7a844121cc631eeebd9dba339a598944d622262ce0e65: Status 404 returned error can't find the container with id 61f079e328f526489be7a844121cc631eeebd9dba339a598944d622262ce0e65
Oct 30 21:26:45.421629 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:45.421596    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl" event={"ID":"94038fcc-f568-4af3-adca-9528d7d65ce5","Type":"ContainerStarted","Data":"61f079e328f526489be7a844121cc631eeebd9dba339a598944d622262ce0e65"}
Oct 30 21:26:45.422878 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:45.422854    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q" event={"ID":"a2a4ac24-81a3-450c-bb62-beefd608d486","Type":"ContainerStarted","Data":"f394ab23771267d32685753165bb095c04e3f3aab35b8b2f349f78872aa54331"}
Oct 30 21:26:46.429388 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:46.429353    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl" event={"ID":"94038fcc-f568-4af3-adca-9528d7d65ce5","Type":"ContainerStarted","Data":"94ad5f6aa689f9dc5be1af3886ef5d0f516a3647d9a1363ee50180d174a87eab"}
Oct 30 21:26:46.429881 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:46.429430    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:46.431046 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:46.431023    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q" event={"ID":"a2a4ac24-81a3-450c-bb62-beefd608d486","Type":"ContainerStarted","Data":"4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc"}
Oct 30 21:26:46.431152 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:46.431141    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:46.548229 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:46.548177    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl" podStartSLOduration=2.737481942 podStartE2EDuration="3.548162779s" totalImagesPullingTime="810.680837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:46.544944928 +0000 UTC m=+4032.693995919" watchObservedRunningTime="2026-10-30 21:26:46.548162779 +0000 UTC m=+4032.697213767"
Oct 30 21:26:46.662294 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:46.662217    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q" podStartSLOduration=2.791074547 podStartE2EDuration="3.662197767s" totalImagesPullingTime="871.12322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:46.660794511 +0000 UTC m=+4032.809845499" watchObservedRunningTime="2026-10-30 21:26:46.662197767 +0000 UTC m=+4032.811248759"
Oct 30 21:26:46.897486 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:46.897443    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:51.198580 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:51.198544    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:52.004361 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.004322    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:52.371959 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.371874    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:52.442976 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.442947    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-4bnbl"
Oct 30 21:26:52.443344 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.443323    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:52.614587 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.614507    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"]
Oct 30 21:26:52.614885 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.614858    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q" podUID="a2a4ac24-81a3-450c-bb62-beefd608d486" containerName="maas-api" containerID="cri-o://4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc" gracePeriod=30
Oct 30 21:26:52.873436 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.873410    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:52.970772 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.970744    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-maas-api-tls\") pod \"a2a4ac24-81a3-450c-bb62-beefd608d486\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") "
Oct 30 21:26:52.970772 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.970776    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-metrics-certs\") pod \"a2a4ac24-81a3-450c-bb62-beefd608d486\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") "
Oct 30 21:26:52.971018 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.970842    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a4ac24-81a3-450c-bb62-beefd608d486-kube-api-access-gsz6w\" (UniqueName: \"kubernetes.io/projected/a2a4ac24-81a3-450c-bb62-beefd608d486-kube-api-access-gsz6w\") pod \"a2a4ac24-81a3-450c-bb62-beefd608d486\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") "
Oct 30 21:26:52.971018 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.970869    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a4ac24-81a3-450c-bb62-beefd608d486-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4ac24-81a3-450c-bb62-beefd608d486-openshift-service-ca\") pod \"a2a4ac24-81a3-450c-bb62-beefd608d486\" (UID: \"a2a4ac24-81a3-450c-bb62-beefd608d486\") "
Oct 30 21:26:52.971331 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.971308    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a4ac24-81a3-450c-bb62-beefd608d486-openshift-service-ca" pod "a2a4ac24-81a3-450c-bb62-beefd608d486" (UID: "a2a4ac24-81a3-450c-bb62-beefd608d486"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:52.973138 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.973114    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a4ac24-81a3-450c-bb62-beefd608d486-kube-api-access-gsz6w" pod "a2a4ac24-81a3-450c-bb62-beefd608d486" (UID: "a2a4ac24-81a3-450c-bb62-beefd608d486"). InnerVolumeSpecName "kube-api-access-gsz6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:52.973260 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.973230    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-metrics-certs" pod "a2a4ac24-81a3-450c-bb62-beefd608d486" (UID: "a2a4ac24-81a3-450c-bb62-beefd608d486"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:52.973379 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:52.973367    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-maas-api-tls" pod "a2a4ac24-81a3-450c-bb62-beefd608d486" (UID: "a2a4ac24-81a3-450c-bb62-beefd608d486"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:53.072384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.072353    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsz6w\" (UniqueName: \"kubernetes.io/projected/a2a4ac24-81a3-450c-bb62-beefd608d486-kube-api-access-gsz6w\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:53.072384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.072379    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4ac24-81a3-450c-bb62-beefd608d486-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:53.072672 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.072466    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:53.072672 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.072481    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ac24-81a3-450c-bb62-beefd608d486-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:53.471222 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.471193    5002 generic.go:356] "Generic (PLEG): container finished" podID="a2a4ac24-81a3-450c-bb62-beefd608d486" containerID="4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc" exitCode=0
Oct 30 21:26:53.471688 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.471232    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"
Oct 30 21:26:53.471688 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.471272    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q" event={"ID":"a2a4ac24-81a3-450c-bb62-beefd608d486","Type":"ContainerDied","Data":"4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc"}
Oct 30 21:26:53.471688 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.471310    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-cws4q" event={"ID":"a2a4ac24-81a3-450c-bb62-beefd608d486","Type":"ContainerDied","Data":"f394ab23771267d32685753165bb095c04e3f3aab35b8b2f349f78872aa54331"}
Oct 30 21:26:53.471688 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.471330    5002 scope.go:122] "RemoveContainer" containerID="4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc"
Oct 30 21:26:53.481321 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.481302    5002 scope.go:122] "RemoveContainer" containerID="4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc"
Oct 30 21:26:53.481596 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:26:53.481566    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc\": container with ID starting with 4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc not found: ID does not exist" containerID="4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc"
Oct 30 21:26:53.481648 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.481600    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc"} err="failed to get container status \"4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc\": rpc error: code = NotFound desc = could not find container \"4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc\": container with ID starting with 4399738953bce06e078bf53641da24aa0fb8b0aa965eab777b7715323574b9cc not found: ID does not exist"
Oct 30 21:26:53.637179 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.637152    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"]
Oct 30 21:26:53.687195 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.687170    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-cws4q"]
Oct 30 21:26:53.965194 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:53.965163    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a4ac24-81a3-450c-bb62-beefd608d486" path="/var/lib/kubelet/pods/a2a4ac24-81a3-450c-bb62-beefd608d486/volumes"
Oct 30 21:26:55.223432 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:55.223400    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:55.481056 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:55.480969    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:55.658786 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:55.658756    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:58.599968 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:58.599906    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:58.774707 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:58.774670    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:59.117787 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.117747    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"]
Oct 30 21:26:59.118608 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.118579    5002 cpu_manager.go:414] "containerMap: removing container" podUID="a2a4ac24-81a3-450c-bb62-beefd608d486" containerName="maas-api"
Oct 30 21:26:59.118608 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.118605    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a4ac24-81a3-450c-bb62-beefd608d486" containerName="maas-api"
Oct 30 21:26:59.118866 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.118733    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a4ac24-81a3-450c-bb62-beefd608d486" containerName="maas-api"
Oct 30 21:26:59.124099 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.124080    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.179407 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.179384    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"]
Oct 30 21:26:59.226797 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.226774    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fb4770d7\""
Oct 30 21:26:59.226797 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.226789    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fb4770d7-dockercfg-b8l4q\""
Oct 30 21:26:59.233318 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.233294    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d3abd-d2e5-4c47-be61-975818c8bd4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") " pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.233431 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.233342    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhjb\" (UniqueName: \"kubernetes.io/projected/e99d3abd-d2e5-4c47-be61-975818c8bd4e-kube-api-access-snhjb\") pod \"payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") " pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.285799 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.285776    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"]
Oct 30 21:26:59.289382 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.289365    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.297618 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.297596    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"]
Oct 30 21:26:59.300954 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.300936    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.334254 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.334234    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d3abd-d2e5-4c47-be61-975818c8bd4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") " pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.334354 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.334269    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhjb\" (UniqueName: \"kubernetes.io/projected/e99d3abd-d2e5-4c47-be61-975818c8bd4e-kube-api-access-snhjb\") pod \"payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") " pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.334901 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.334885    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d3abd-d2e5-4c47-be61-975818c8bd4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") " pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.344362 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.344345    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fb4770d7-serving-cert\""
Oct 30 21:26:59.347809 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.347789    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"]
Oct 30 21:26:59.357719 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.357696    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"]
Oct 30 21:26:59.391323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.391271    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhjb\" (UniqueName: \"kubernetes.io/projected/e99d3abd-d2e5-4c47-be61-975818c8bd4e-kube-api-access-snhjb\") pod \"payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") " pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.435717 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.435694    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-metrics-certs\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.435852 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.435750    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4ae330-05d7-4784-854b-970d566a701e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.435928 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.435870    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblkr\" (UniqueName: \"kubernetes.io/projected/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-kube-api-access-cblkr\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.435928 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.435918    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsml\" (UniqueName: \"kubernetes.io/projected/fa4ae330-05d7-4784-854b-970d566a701e-kube-api-access-qcsml\") pod \"payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.436038 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.435944    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-openshift-service-ca\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.436078 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.436064    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.437545 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.437506    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:26:59.526663 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.526625    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:59.538207 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.538171    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.538806 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.538781    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-metrics-certs\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.539505 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.539484    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4ae330-05d7-4784-854b-970d566a701e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.541007 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:26:59.538642    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-serving-cert: secret "maas-api-e2e-ait-fb4770d7-serving-cert" not found
Oct 30 21:26:59.541007 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:26:59.540676    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls podName:3a24badb-78d8-45eb-909d-5cf2e2a20ac1 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:00.040654881 +0000 UTC m=+4046.189705848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls") pod "maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x" (UID: "3a24badb-78d8-45eb-909d-5cf2e2a20ac1") : secret "maas-api-e2e-ait-fb4770d7-serving-cert" not found
Oct 30 21:26:59.541007 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.540486    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4ae330-05d7-4784-854b-970d566a701e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.541549 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.540588    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblkr\" (UniqueName: \"kubernetes.io/projected/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-kube-api-access-cblkr\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.541746 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.541726    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsml\" (UniqueName: \"kubernetes.io/projected/fa4ae330-05d7-4784-854b-970d566a701e-kube-api-access-qcsml\") pod \"payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.542104 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.542082    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-openshift-service-ca\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.543074 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.542942    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-openshift-service-ca\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.545641 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.545586    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-metrics-certs\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.601303 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.601274    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsml\" (UniqueName: \"kubernetes.io/projected/fa4ae330-05d7-4784-854b-970d566a701e-kube-api-access-qcsml\") pod \"payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.601624 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.601305    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblkr\" (UniqueName: \"kubernetes.io/projected/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-kube-api-access-cblkr\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:26:59.612889 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.612871    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:26:59.702929 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.702894    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:59.812118 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.811946    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"]
Oct 30 21:26:59.813341 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:26:59.813309    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99d3abd_d2e5_4c47_be61_975818c8bd4e.slice/crio-bc7b01a3f73c49b760a39df3ec764e72b5e3305394ad4536956f807668504c6d WatchSource:0}: Error finding container bc7b01a3f73c49b760a39df3ec764e72b5e3305394ad4536956f807668504c6d: Status 404 returned error can't find the container with id bc7b01a3f73c49b760a39df3ec764e72b5e3305394ad4536956f807668504c6d
Oct 30 21:26:59.879729 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.879692    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:26:59.986216 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:26:59.986190    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"]
Oct 30 21:26:59.987291 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:26:59.987269    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4ae330_05d7_4784_854b_970d566a701e.slice/crio-2c34e9b21a5ecdee95d4873c1c9c6f6a71e0f4e2195c73b49946fa55a9c7a1b4 WatchSource:0}: Error finding container 2c34e9b21a5ecdee95d4873c1c9c6f6a71e0f4e2195c73b49946fa55a9c7a1b4: Status 404 returned error can't find the container with id 2c34e9b21a5ecdee95d4873c1c9c6f6a71e0f4e2195c73b49946fa55a9c7a1b4
Oct 30 21:27:00.048827 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.048806    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:27:00.051081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.051060    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls\") pod \"maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:27:00.202556 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.202506    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:27:00.516751 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.516677    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg" event={"ID":"e99d3abd-d2e5-4c47-be61-975818c8bd4e","Type":"ContainerStarted","Data":"67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7"}
Oct 30 21:27:00.516751 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.516731    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg" event={"ID":"e99d3abd-d2e5-4c47-be61-975818c8bd4e","Type":"ContainerStarted","Data":"bc7b01a3f73c49b760a39df3ec764e72b5e3305394ad4536956f807668504c6d"}
Oct 30 21:27:00.517349 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.517319    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:27:00.526660 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.526623    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv" event={"ID":"fa4ae330-05d7-4784-854b-970d566a701e","Type":"ContainerStarted","Data":"01940ab8c22ac6a88aa6f7d78f9854c89bf825f3e90fd5b343a4941b4441f612"}
Oct 30 21:27:00.526781 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.526717    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv" event={"ID":"fa4ae330-05d7-4784-854b-970d566a701e","Type":"ContainerStarted","Data":"2c34e9b21a5ecdee95d4873c1c9c6f6a71e0f4e2195c73b49946fa55a9c7a1b4"}
Oct 30 21:27:00.623036 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.623008    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"]
Oct 30 21:27:00.623866 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:27:00.623847    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a24badb_78d8_45eb_909d_5cf2e2a20ac1.slice/crio-834891e0dbb1e2618b25641950fc5cd3bc78c8041eeb1240723c53dcd4318874 WatchSource:0}: Error finding container 834891e0dbb1e2618b25641950fc5cd3bc78c8041eeb1240723c53dcd4318874: Status 404 returned error can't find the container with id 834891e0dbb1e2618b25641950fc5cd3bc78c8041eeb1240723c53dcd4318874
Oct 30 21:27:00.635248 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.635204    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg" podStartSLOduration=1.635192824 podStartE2EDuration="1.635192824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:00.633225048 +0000 UTC m=+4046.782276035" watchObservedRunningTime="2026-10-30 21:27:00.635192824 +0000 UTC m=+4046.784243812"
Oct 30 21:27:00.755698 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:00.755635    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv" podStartSLOduration=1.7556206680000002 podStartE2EDuration="1.755620668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:00.752577508 +0000 UTC m=+4046.901628499" watchObservedRunningTime="2026-10-30 21:27:00.755620668 +0000 UTC m=+4046.904671656"
Oct 30 21:27:01.534193 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:01.534155    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x" event={"ID":"3a24badb-78d8-45eb-909d-5cf2e2a20ac1","Type":"ContainerStarted","Data":"a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674"}
Oct 30 21:27:01.534334 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:01.534210    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x" event={"ID":"3a24badb-78d8-45eb-909d-5cf2e2a20ac1","Type":"ContainerStarted","Data":"834891e0dbb1e2618b25641950fc5cd3bc78c8041eeb1240723c53dcd4318874"}
Oct 30 21:27:01.534334 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:01.534287    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:27:01.534602 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:01.534586    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:27:01.656300 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:01.656248    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x" podStartSLOduration=1.84885656 podStartE2EDuration="2.656233986s" totalImagesPullingTime="807.377426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:01.655471187 +0000 UTC m=+4047.804522176" watchObservedRunningTime="2026-10-30 21:27:01.656233986 +0000 UTC m=+4047.805284974"
Oct 30 21:27:04.845080 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:04.845027    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:07.116026 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:27:07.115983    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:27:07.514672 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:07.514630    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:27:07.545248 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:07.545222    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:27:09.757483 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:09.757439    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:11.537897 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:11.537870    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:27:12.543327 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:12.543288    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:27:14.651330 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:14.651291    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:19.856887 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:19.856836    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:24.965750 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:24.965716    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:29.820720 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:29.820684    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:34.624565 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:34.624508    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:37.114344 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:27:37.114298    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:27:39.430925 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:39.430890    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:43.457958 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:43.457882    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"]
Oct 30 21:27:43.565962 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:43.565933    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"]
Oct 30 21:27:43.855259 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:43.855224    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:44.031340 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:44.031304    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:44.221521 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:44.221495    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:47.248631 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:47.248599    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:48.299785 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.299747    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"]
Oct 30 21:27:48.300266 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.299986    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x" podUID="3a24badb-78d8-45eb-909d-5cf2e2a20ac1" containerName="maas-api" containerID="cri-o://a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674" gracePeriod=30
Oct 30 21:27:48.444887 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.444845    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:48.458754 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.458713    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg" podUID="e99d3abd-d2e5-4c47-be61-975818c8bd4e" containerName="payload-processing" containerID="cri-o://67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7" gracePeriod=25
Oct 30 21:27:48.566913 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.566878    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv" podUID="fa4ae330-05d7-4784-854b-970d566a701e" containerName="payload-pre-processing" containerID="cri-o://01940ab8c22ac6a88aa6f7d78f9854c89bf825f3e90fd5b343a4941b4441f612" gracePeriod=25
Oct 30 21:27:48.636883 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.636854    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:27:48.712218 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.712194    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:27:48.781224 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781194    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-kube-api-access-cblkr\" (UniqueName: \"kubernetes.io/projected/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-kube-api-access-cblkr\") pod \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") "
Oct 30 21:27:48.781400 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781265    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99d3abd-d2e5-4c47-be61-975818c8bd4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d3abd-d2e5-4c47-be61-975818c8bd4e-plugins-config-volume\") pod \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") "
Oct 30 21:27:48.781400 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781300    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99d3abd-d2e5-4c47-be61-975818c8bd4e-kube-api-access-snhjb\" (UniqueName: \"kubernetes.io/projected/e99d3abd-d2e5-4c47-be61-975818c8bd4e-kube-api-access-snhjb\") pod \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\" (UID: \"e99d3abd-d2e5-4c47-be61-975818c8bd4e\") "
Oct 30 21:27:48.781400 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781370    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-openshift-service-ca\") pod \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") "
Oct 30 21:27:48.781607 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781427    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls\") pod \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") "
Oct 30 21:27:48.781607 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781468    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-metrics-certs\") pod \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\" (UID: \"3a24badb-78d8-45eb-909d-5cf2e2a20ac1\") "
Oct 30 21:27:48.781786 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781723    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99d3abd-d2e5-4c47-be61-975818c8bd4e-plugins-config-volume" pod "e99d3abd-d2e5-4c47-be61-975818c8bd4e" (UID: "e99d3abd-d2e5-4c47-be61-975818c8bd4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:48.781883 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.781861    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d3abd-d2e5-4c47-be61-975818c8bd4e-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:48.782109 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.782032    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-openshift-service-ca" pod "3a24badb-78d8-45eb-909d-5cf2e2a20ac1" (UID: "3a24badb-78d8-45eb-909d-5cf2e2a20ac1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:48.783593 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.783564    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-kube-api-access-cblkr" pod "3a24badb-78d8-45eb-909d-5cf2e2a20ac1" (UID: "3a24badb-78d8-45eb-909d-5cf2e2a20ac1"). InnerVolumeSpecName "kube-api-access-cblkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:48.783848 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.783826    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls" pod "3a24badb-78d8-45eb-909d-5cf2e2a20ac1" (UID: "3a24badb-78d8-45eb-909d-5cf2e2a20ac1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:27:48.783983 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.783964    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-metrics-certs" pod "3a24badb-78d8-45eb-909d-5cf2e2a20ac1" (UID: "3a24badb-78d8-45eb-909d-5cf2e2a20ac1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:27:48.784085 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.784069    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99d3abd-d2e5-4c47-be61-975818c8bd4e-kube-api-access-snhjb" pod "e99d3abd-d2e5-4c47-be61-975818c8bd4e" (UID: "e99d3abd-d2e5-4c47-be61-975818c8bd4e"). InnerVolumeSpecName "kube-api-access-snhjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:48.798972 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.798942    5002 generic.go:356] "Generic (PLEG): container finished" podID="e99d3abd-d2e5-4c47-be61-975818c8bd4e" containerID="67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7" exitCode=0
Oct 30 21:27:48.799084 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.799001    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"
Oct 30 21:27:48.799084 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.799022    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg" event={"ID":"e99d3abd-d2e5-4c47-be61-975818c8bd4e","Type":"ContainerDied","Data":"67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7"}
Oct 30 21:27:48.799217 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.799092    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg" event={"ID":"e99d3abd-d2e5-4c47-be61-975818c8bd4e","Type":"ContainerDied","Data":"bc7b01a3f73c49b760a39df3ec764e72b5e3305394ad4536956f807668504c6d"}
Oct 30 21:27:48.799217 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.799123    5002 scope.go:122] "RemoveContainer" containerID="67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7"
Oct 30 21:27:48.801211 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.801189    5002 generic.go:356] "Generic (PLEG): container finished" podID="fa4ae330-05d7-4784-854b-970d566a701e" containerID="01940ab8c22ac6a88aa6f7d78f9854c89bf825f3e90fd5b343a4941b4441f612" exitCode=0
Oct 30 21:27:48.801322 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.801272    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv" event={"ID":"fa4ae330-05d7-4784-854b-970d566a701e","Type":"ContainerDied","Data":"01940ab8c22ac6a88aa6f7d78f9854c89bf825f3e90fd5b343a4941b4441f612"}
Oct 30 21:27:48.802694 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.802671    5002 generic.go:356] "Generic (PLEG): container finished" podID="3a24badb-78d8-45eb-909d-5cf2e2a20ac1" containerID="a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674" exitCode=0
Oct 30 21:27:48.802797 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.802723    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"
Oct 30 21:27:48.802797 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.802751    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x" event={"ID":"3a24badb-78d8-45eb-909d-5cf2e2a20ac1","Type":"ContainerDied","Data":"a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674"}
Oct 30 21:27:48.802797 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.802781    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x" event={"ID":"3a24badb-78d8-45eb-909d-5cf2e2a20ac1","Type":"ContainerDied","Data":"834891e0dbb1e2618b25641950fc5cd3bc78c8041eeb1240723c53dcd4318874"}
Oct 30 21:27:48.811017 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.810893    5002 scope.go:122] "RemoveContainer" containerID="67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7"
Oct 30 21:27:48.812684 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:27:48.812661    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7\": container with ID starting with 67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7 not found: ID does not exist" containerID="67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7"
Oct 30 21:27:48.812787 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.812695    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7"} err="failed to get container status \"67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7\": rpc error: code = NotFound desc = could not find container \"67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7\": container with ID starting with 67d2af96df295e5920a24e2150aee234dea26bde590d774ca02ed6d1355dd6a7 not found: ID does not exist"
Oct 30 21:27:48.812787 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.812717    5002 scope.go:122] "RemoveContainer" containerID="a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674"
Oct 30 21:27:48.822243 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.822229    5002 scope.go:122] "RemoveContainer" containerID="a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674"
Oct 30 21:27:48.822512 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:27:48.822495    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674\": container with ID starting with a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674 not found: ID does not exist" containerID="a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674"
Oct 30 21:27:48.822585 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.822521    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674"} err="failed to get container status \"a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674\": rpc error: code = NotFound desc = could not find container \"a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674\": container with ID starting with a730ad9aa078518eeca532f66248b870a23a3b7cfff99a652e395ed51aeb7674 not found: ID does not exist"
Oct 30 21:27:48.839517 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.839501    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:27:48.884181 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.884147    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:48.884181 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.884181    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:48.884414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.884196    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblkr\" (UniqueName: \"kubernetes.io/projected/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-kube-api-access-cblkr\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:48.884414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.884211    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhjb\" (UniqueName: \"kubernetes.io/projected/e99d3abd-d2e5-4c47-be61-975818c8bd4e-kube-api-access-snhjb\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:48.884414 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.884226    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a24badb-78d8-45eb-909d-5cf2e2a20ac1-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:48.985427 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.985398    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4ae330-05d7-4784-854b-970d566a701e-kube-api-access-qcsml\" (UniqueName: \"kubernetes.io/projected/fa4ae330-05d7-4784-854b-970d566a701e-kube-api-access-qcsml\") pod \"fa4ae330-05d7-4784-854b-970d566a701e\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") "
Oct 30 21:27:48.985636 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.985433    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4ae330-05d7-4784-854b-970d566a701e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4ae330-05d7-4784-854b-970d566a701e-plugins-config-volume\") pod \"fa4ae330-05d7-4784-854b-970d566a701e\" (UID: \"fa4ae330-05d7-4784-854b-970d566a701e\") "
Oct 30 21:27:48.985887 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.985865    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4ae330-05d7-4784-854b-970d566a701e-plugins-config-volume" pod "fa4ae330-05d7-4784-854b-970d566a701e" (UID: "fa4ae330-05d7-4784-854b-970d566a701e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:48.987576 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.987527    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4ae330-05d7-4784-854b-970d566a701e-kube-api-access-qcsml" pod "fa4ae330-05d7-4784-854b-970d566a701e" (UID: "fa4ae330-05d7-4784-854b-970d566a701e"). InnerVolumeSpecName "kube-api-access-qcsml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:48.992291 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.992269    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"]
Oct 30 21:27:48.994556 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:48.994522    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg"]
Oct 30 21:27:49.086357 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.086307    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsml\" (UniqueName: \"kubernetes.io/projected/fa4ae330-05d7-4784-854b-970d566a701e-kube-api-access-qcsml\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:49.086357 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.086328    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4ae330-05d7-4784-854b-970d566a701e-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:49.104581 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.104552    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"]
Oct 30 21:27:49.109238 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.109219    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb4770d7-6b845f9ffd-lmz4x"]
Oct 30 21:27:49.811965 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.811940    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"
Oct 30 21:27:49.812384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.811966    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv" event={"ID":"fa4ae330-05d7-4784-854b-970d566a701e","Type":"ContainerDied","Data":"2c34e9b21a5ecdee95d4873c1c9c6f6a71e0f4e2195c73b49946fa55a9c7a1b4"}
Oct 30 21:27:49.812384 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.812036    5002 scope.go:122] "RemoveContainer" containerID="01940ab8c22ac6a88aa6f7d78f9854c89bf825f3e90fd5b343a4941b4441f612"
Oct 30 21:27:49.965819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.965792    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a24badb-78d8-45eb-909d-5cf2e2a20ac1" path="/var/lib/kubelet/pods/3a24badb-78d8-45eb-909d-5cf2e2a20ac1/volumes"
Oct 30 21:27:49.966170 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.966157    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99d3abd-d2e5-4c47-be61-975818c8bd4e" path="/var/lib/kubelet/pods/e99d3abd-d2e5-4c47-be61-975818c8bd4e/volumes"
Oct 30 21:27:49.977138 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.977114    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"]
Oct 30 21:27:49.983572 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:49.983549    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv"]
Oct 30 21:27:50.007071 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:50.007048    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:50.288383 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:50.288349    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:51.965386 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:51.965352    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4ae330-05d7-4784-854b-970d566a701e" path="/var/lib/kubelet/pods/fa4ae330-05d7-4784-854b-970d566a701e/volumes"
Oct 30 21:27:52.770422 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:52.770392    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:27:54.189739 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:54.189706    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:55.535953 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:55.535917    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:56.153867 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:56.153831    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:27:56.362862 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:27:56.362819    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:00.141261 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:00.141221    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:00.330003 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:00.329969    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:00.524828 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:00.524790    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:03.377366 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.377334    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:03.557349 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.557299    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:03.760752 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.760723    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"]
Oct 30 21:28:03.761177 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761164    5002 cpu_manager.go:414] "containerMap: removing container" podUID="fa4ae330-05d7-4784-854b-970d566a701e" containerName="payload-pre-processing"
Oct 30 21:28:03.761222 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761178    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4ae330-05d7-4784-854b-970d566a701e" containerName="payload-pre-processing"
Oct 30 21:28:03.761222 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761195    5002 cpu_manager.go:414] "containerMap: removing container" podUID="3a24badb-78d8-45eb-909d-5cf2e2a20ac1" containerName="maas-api"
Oct 30 21:28:03.761222 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761199    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a24badb-78d8-45eb-909d-5cf2e2a20ac1" containerName="maas-api"
Oct 30 21:28:03.761323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761226    5002 cpu_manager.go:414] "containerMap: removing container" podUID="e99d3abd-d2e5-4c47-be61-975818c8bd4e" containerName="payload-processing"
Oct 30 21:28:03.761323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761233    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99d3abd-d2e5-4c47-be61-975818c8bd4e" containerName="payload-processing"
Oct 30 21:28:03.761323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761294    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99d3abd-d2e5-4c47-be61-975818c8bd4e" containerName="payload-processing"
Oct 30 21:28:03.761323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761304    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4ae330-05d7-4784-854b-970d566a701e" containerName="payload-pre-processing"
Oct 30 21:28:03.761323 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.761311    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a24badb-78d8-45eb-909d-5cf2e2a20ac1" containerName="maas-api"
Oct 30 21:28:03.765811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.765792    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:03.821823 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.821790    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"]
Oct 30 21:28:03.869813 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.869784    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-0a3a62\""
Oct 30 21:28:03.869991 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.869883    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-0a3a62-dockercfg-qw8pr\""
Oct 30 21:28:03.922912 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.922881    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667d8748-f305-4838-a6a2-d6b45bc6d821-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:03.923106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.922992    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv42q\" (UniqueName: \"kubernetes.io/projected/667d8748-f305-4838-a6a2-d6b45bc6d821-kube-api-access-sv42q\") pod \"payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:03.923106 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.923053    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"]
Oct 30 21:28:03.932885 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.932850    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:03.966691 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:03.966663    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"]
Oct 30 21:28:04.024527 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.024454    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:04.024701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.024600    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667d8748-f305-4838-a6a2-d6b45bc6d821-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:04.024701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.024641    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7c7f\" (UniqueName: \"kubernetes.io/projected/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-kube-api-access-k7c7f\") pod \"payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:04.024701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.024692    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv42q\" (UniqueName: \"kubernetes.io/projected/667d8748-f305-4838-a6a2-d6b45bc6d821-kube-api-access-sv42q\") pod \"payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:04.025324 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.025299    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667d8748-f305-4838-a6a2-d6b45bc6d821-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:04.081958 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.081935    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv42q\" (UniqueName: \"kubernetes.io/projected/667d8748-f305-4838-a6a2-d6b45bc6d821-kube-api-access-sv42q\") pod \"payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:04.125741 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.125718    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7c7f\" (UniqueName: \"kubernetes.io/projected/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-kube-api-access-k7c7f\") pod \"payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:04.125839 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.125799    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:04.126394 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.126378    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:04.183786 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.183761    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7c7f\" (UniqueName: \"kubernetes.io/projected/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-kube-api-access-k7c7f\") pod \"payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:04.245854 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.245831    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:04.378891 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.378855    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:04.617903 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.617877    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"]
Oct 30 21:28:04.619217 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:28:04.619194    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0632c00e_df64_4a76_8fa7_dbb9cb09ed00.slice/crio-61b86735368f01da02ce37a3ccb1ebbe239ec922578bf550d59bf7f4b3d1f9b9 WatchSource:0}: Error finding container 61b86735368f01da02ce37a3ccb1ebbe239ec922578bf550d59bf7f4b3d1f9b9: Status 404 returned error can't find the container with id 61b86735368f01da02ce37a3ccb1ebbe239ec922578bf550d59bf7f4b3d1f9b9
Oct 30 21:28:04.750495 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.750474    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"]
Oct 30 21:28:04.751186 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:28:04.751166    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667d8748_f305_4838_a6a2_d6b45bc6d821.slice/crio-ef67b5bea0627d0c7a4a4ef2a26ee60a7b4aa18cb34ca781cb25198362ccc5ef WatchSource:0}: Error finding container ef67b5bea0627d0c7a4a4ef2a26ee60a7b4aa18cb34ca781cb25198362ccc5ef: Status 404 returned error can't find the container with id ef67b5bea0627d0c7a4a4ef2a26ee60a7b4aa18cb34ca781cb25198362ccc5ef
Oct 30 21:28:04.903069 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.902991    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl" event={"ID":"667d8748-f305-4838-a6a2-d6b45bc6d821","Type":"ContainerStarted","Data":"556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03"}
Oct 30 21:28:04.903069 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.903051    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl" event={"ID":"667d8748-f305-4838-a6a2-d6b45bc6d821","Type":"ContainerStarted","Data":"ef67b5bea0627d0c7a4a4ef2a26ee60a7b4aa18cb34ca781cb25198362ccc5ef"}
Oct 30 21:28:04.903269 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.903078    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:04.904896 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.904867    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v" event={"ID":"0632c00e-df64-4a76-8fa7-dbb9cb09ed00","Type":"ContainerStarted","Data":"4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25"}
Oct 30 21:28:04.905043 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.904918    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v" event={"ID":"0632c00e-df64-4a76-8fa7-dbb9cb09ed00","Type":"ContainerStarted","Data":"61b86735368f01da02ce37a3ccb1ebbe239ec922578bf550d59bf7f4b3d1f9b9"}
Oct 30 21:28:04.905043 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:04.904947    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:05.018356 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:05.018315    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl" podStartSLOduration=2.018303248 podStartE2EDuration="2.018303248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:05.017907829 +0000 UTC m=+4111.166958817" watchObservedRunningTime="2026-10-30 21:28:05.018303248 +0000 UTC m=+4111.167354234"
Oct 30 21:28:05.136935 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:05.136867    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v" podStartSLOduration=2.136849441 podStartE2EDuration="2.136849441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:05.132566484 +0000 UTC m=+4111.281617471" watchObservedRunningTime="2026-10-30 21:28:05.136849441 +0000 UTC m=+4111.285900429"
Oct 30 21:28:06.135504 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.135470    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"]
Oct 30 21:28:06.139105 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.139088    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.244883 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.244855    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-0a3a62-serving-cert\""
Oct 30 21:28:06.246168 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.246145    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzd4\" (UniqueName: \"kubernetes.io/projected/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-kube-api-access-pfzd4\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.246271 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.246255    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.246306 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.246292    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.246387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.246374    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.247431 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.247411    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"]
Oct 30 21:28:06.347781 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.347742    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.347924 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.347817    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzd4\" (UniqueName: \"kubernetes.io/projected/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-kube-api-access-pfzd4\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.347924 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.347894    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.347924 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:28:06.347903    5002 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-serving-cert: secret "maas-api-e2e-authcfg-http-0a3a62-serving-cert" not found
Oct 30 21:28:06.348050 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.347928    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.348050 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:28:06.347972    5002 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls podName:4f5d66fa-39c2-4e5f-b2c7-c1253827ec03 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:06.847952214 +0000 UTC m=+4112.997003179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls") pod "maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn" (UID: "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03") : secret "maas-api-e2e-authcfg-http-0a3a62-serving-cert" not found
Oct 30 21:28:06.348518 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.348496    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.350502 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.350481    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.372326 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.372303    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:06.409206 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.409148    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzd4\" (UniqueName: \"kubernetes.io/projected/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-kube-api-access-pfzd4\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.550576 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.550496    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:06.852880 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.852846    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:06.855395 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:06.855375    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:07.053024 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:07.052996    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:07.127138 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:28:07.127102    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:28:07.428422 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:28:07.428396    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5d66fa_39c2_4e5f_b2c7_c1253827ec03.slice/crio-683dc22aa19fe5c967ec324f420f0b2296103db931732e4c47d068d4813d3ee1 WatchSource:0}: Error finding container 683dc22aa19fe5c967ec324f420f0b2296103db931732e4c47d068d4813d3ee1: Status 404 returned error can't find the container with id 683dc22aa19fe5c967ec324f420f0b2296103db931732e4c47d068d4813d3ee1
Oct 30 21:28:07.453819 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:07.453785    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"]
Oct 30 21:28:07.922595 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:07.922559    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn" event={"ID":"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03","Type":"ContainerStarted","Data":"683dc22aa19fe5c967ec324f420f0b2296103db931732e4c47d068d4813d3ee1"}
Oct 30 21:28:08.929863 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:08.929815    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn" event={"ID":"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03","Type":"ContainerStarted","Data":"bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7"}
Oct 30 21:28:08.930257 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:08.929917    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:09.054178 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:09.054131    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn" podStartSLOduration=2.240930057 podStartE2EDuration="3.054118021s" totalImagesPullingTime="813.187964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:09.049871458 +0000 UTC m=+4115.198922449" watchObservedRunningTime="2026-10-30 21:28:09.054118021 +0000 UTC m=+4115.203169007"
Oct 30 21:28:11.325898 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:11.325811    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:13.116551 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:13.116504    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:28:14.940576 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:14.940548    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:28:15.912688 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:15.912657    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:28:15.913081 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:15.913060    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:28:16.301171 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:16.301129    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:21.123595 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:21.123559    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:25.031931 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:25.031888    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:26.376815 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:26.376773    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:31.212591 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:31.212553    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:36.021067 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:36.021032    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:37.114351 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:28:37.114319    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:28:40.408065 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:40.408035    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:40.835276 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:40.835236    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:45.666889 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:45.666850    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:47.566992 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:47.566959    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:50.420786 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:50.420752    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:28:55.341600 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:28:55.341522    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:00.131459 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:00.131418    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:05.013197 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:05.013160    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:07.114460 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:29:07.114420    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:29:09.846045 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:09.845995    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:14.729973 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:14.729883    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:19.529416 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:19.529381    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:24.328925 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:24.328880    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:29.122057 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:29.122026    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:33.934289 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:33.934261    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:34.605696 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:34.605668    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:29:34.630928 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:34.630908    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jnzz2_7b7f4d68-a578-434f-894a-6cf2fbc63cda/console-operator/2.log"
Oct 30 21:29:37.114106 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:29:37.114070    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:29:38.743826 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:38.743787    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:43.628914 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:43.628867    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:47.915947 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:47.915913    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:48.883684 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:48.883650    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:53.739002 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:53.738966    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:54.886379 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:54.886341    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:57.155820 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:57.155789    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"]
Oct 30 21:29:57.262898 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:57.262869    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"]
Oct 30 21:29:57.558491 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:57.558458    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:57.742254 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:57.742224    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:29:58.533862 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:29:58.533826    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:02.157002 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.156964    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v" podUID="0632c00e-df64-4a76-8fa7-dbb9cb09ed00" containerName="payload-processing" containerID="cri-o://4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25" gracePeriod=25
Oct 30 21:30:02.263320 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.263288    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl" podUID="667d8748-f305-4838-a6a2-d6b45bc6d821" containerName="payload-pre-processing" containerID="cri-o://556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03" gracePeriod=25
Oct 30 21:30:02.416661 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.416573    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:30:02.463204 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.463173    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-plugins-config-volume\") pod \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") "
Oct 30 21:30:02.463379 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.463353    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-kube-api-access-k7c7f\" (UniqueName: \"kubernetes.io/projected/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-kube-api-access-k7c7f\") pod \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\" (UID: \"0632c00e-df64-4a76-8fa7-dbb9cb09ed00\") "
Oct 30 21:30:02.463571 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.463527    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-plugins-config-volume" pod "0632c00e-df64-4a76-8fa7-dbb9cb09ed00" (UID: "0632c00e-df64-4a76-8fa7-dbb9cb09ed00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:02.463712 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.463695    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:02.466393 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.466362    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-kube-api-access-k7c7f" pod "0632c00e-df64-4a76-8fa7-dbb9cb09ed00" (UID: "0632c00e-df64-4a76-8fa7-dbb9cb09ed00"). InnerVolumeSpecName "kube-api-access-k7c7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:02.514783 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.514761    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:30:02.551890 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.551857    5002 generic.go:356] "Generic (PLEG): container finished" podID="667d8748-f305-4838-a6a2-d6b45bc6d821" containerID="556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03" exitCode=0
Oct 30 21:30:02.552041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.551911    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"
Oct 30 21:30:02.552041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.551948    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl" event={"ID":"667d8748-f305-4838-a6a2-d6b45bc6d821","Type":"ContainerDied","Data":"556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03"}
Oct 30 21:30:02.552041 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.552026    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl" event={"ID":"667d8748-f305-4838-a6a2-d6b45bc6d821","Type":"ContainerDied","Data":"ef67b5bea0627d0c7a4a4ef2a26ee60a7b4aa18cb34ca781cb25198362ccc5ef"}
Oct 30 21:30:02.552234 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.552051    5002 scope.go:122] "RemoveContainer" containerID="556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03"
Oct 30 21:30:02.553682 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.553658    5002 generic.go:356] "Generic (PLEG): container finished" podID="0632c00e-df64-4a76-8fa7-dbb9cb09ed00" containerID="4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25" exitCode=0
Oct 30 21:30:02.553811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.553697    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"
Oct 30 21:30:02.553811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.553733    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v" event={"ID":"0632c00e-df64-4a76-8fa7-dbb9cb09ed00","Type":"ContainerDied","Data":"4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25"}
Oct 30 21:30:02.553811 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.553807    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v" event={"ID":"0632c00e-df64-4a76-8fa7-dbb9cb09ed00","Type":"ContainerDied","Data":"61b86735368f01da02ce37a3ccb1ebbe239ec922578bf550d59bf7f4b3d1f9b9"}
Oct 30 21:30:02.562607 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.562587    5002 scope.go:122] "RemoveContainer" containerID="556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03"
Oct 30 21:30:02.562851 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:30:02.562833    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03\": container with ID starting with 556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03 not found: ID does not exist" containerID="556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03"
Oct 30 21:30:02.562933 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.562860    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03"} err="failed to get container status \"556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03\": rpc error: code = NotFound desc = could not find container \"556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03\": container with ID starting with 556b62e3d96ba2795d09f4737debf0597ac849f8eecabb2285aae286245bad03 not found: ID does not exist"
Oct 30 21:30:02.562933 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.562879    5002 scope.go:122] "RemoveContainer" containerID="4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25"
Oct 30 21:30:02.565091 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.565069    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667d8748-f305-4838-a6a2-d6b45bc6d821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667d8748-f305-4838-a6a2-d6b45bc6d821-plugins-config-volume\") pod \"667d8748-f305-4838-a6a2-d6b45bc6d821\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") "
Oct 30 21:30:02.565213 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.565193    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667d8748-f305-4838-a6a2-d6b45bc6d821-kube-api-access-sv42q\" (UniqueName: \"kubernetes.io/projected/667d8748-f305-4838-a6a2-d6b45bc6d821-kube-api-access-sv42q\") pod \"667d8748-f305-4838-a6a2-d6b45bc6d821\" (UID: \"667d8748-f305-4838-a6a2-d6b45bc6d821\") "
Oct 30 21:30:02.565432 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.565413    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667d8748-f305-4838-a6a2-d6b45bc6d821-plugins-config-volume" pod "667d8748-f305-4838-a6a2-d6b45bc6d821" (UID: "667d8748-f305-4838-a6a2-d6b45bc6d821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:02.565605 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.565588    5002 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667d8748-f305-4838-a6a2-d6b45bc6d821-plugins-config-volume\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:02.565668 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.565612    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7c7f\" (UniqueName: \"kubernetes.io/projected/0632c00e-df64-4a76-8fa7-dbb9cb09ed00-kube-api-access-k7c7f\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:02.567372 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.567350    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667d8748-f305-4838-a6a2-d6b45bc6d821-kube-api-access-sv42q" pod "667d8748-f305-4838-a6a2-d6b45bc6d821" (UID: "667d8748-f305-4838-a6a2-d6b45bc6d821"). InnerVolumeSpecName "kube-api-access-sv42q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:02.571216 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.571195    5002 scope.go:122] "RemoveContainer" containerID="4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25"
Oct 30 21:30:02.571464 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:30:02.571443    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25\": container with ID starting with 4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25 not found: ID does not exist" containerID="4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25"
Oct 30 21:30:02.571518 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.571470    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25"} err="failed to get container status \"4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25\": rpc error: code = NotFound desc = could not find container \"4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25\": container with ID starting with 4957f26d22a1dc6815563c06b155ea380fb9895a6f331e13810c7abb59c3fe25 not found: ID does not exist"
Oct 30 21:30:02.666123 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.666095    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv42q\" (UniqueName: \"kubernetes.io/projected/667d8748-f305-4838-a6a2-d6b45bc6d821-kube-api-access-sv42q\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:02.763482 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.763453    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"]
Oct 30 21:30:02.767049 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.767024    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v"]
Oct 30 21:30:02.972592 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.972566    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"]
Oct 30 21:30:02.977844 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:02.977818    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl"]
Oct 30 21:30:03.964484 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:03.964452    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0632c00e-df64-4a76-8fa7-dbb9cb09ed00" path="/var/lib/kubelet/pods/0632c00e-df64-4a76-8fa7-dbb9cb09ed00/volumes"
Oct 30 21:30:03.964901 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:03.964823    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667d8748-f305-4838-a6a2-d6b45bc6d821" path="/var/lib/kubelet/pods/667d8748-f305-4838-a6a2-d6b45bc6d821/volumes"
Oct 30 21:30:07.114234 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:30:07.114199    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:30:07.660413 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:07.660384    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"]
Oct 30 21:30:07.660624 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:07.660600    5002 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn" podUID="4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" containerName="maas-api" containerID="cri-o://bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7" gracePeriod=30
Oct 30 21:30:07.803211 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:07.803187    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:07.922175 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:07.922119    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:30:08.014893 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.014863    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-openshift-service-ca\") pod \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") "
Oct 30 21:30:08.015028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.014923    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-metrics-certs\") pod \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") "
Oct 30 21:30:08.015028 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.014996    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-kube-api-access-pfzd4\" (UniqueName: \"kubernetes.io/projected/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-kube-api-access-pfzd4\") pod \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") "
Oct 30 21:30:08.015141 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.015066    5002 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls\") pod \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\" (UID: \"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03\") "
Oct 30 21:30:08.015247 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.015221    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-openshift-service-ca" pod "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" (UID: "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:08.015418 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.015393    5002 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-openshift-service-ca\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:08.017075 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.017052    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls" pod "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" (UID: "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:08.017166 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.017112    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-metrics-certs" pod "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" (UID: "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:08.017166 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.017120    5002 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-kube-api-access-pfzd4" pod "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" (UID: "4f5d66fa-39c2-4e5f-b2c7-c1253827ec03"). InnerVolumeSpecName "kube-api-access-pfzd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:08.116491 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.116444    5002 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfzd4\" (UniqueName: \"kubernetes.io/projected/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-kube-api-access-pfzd4\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:08.116903 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.116499    5002 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-maas-api-tls\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:08.116903 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.116521    5002 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03-metrics-certs\") on node \"ip-10-0-141-29.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:08.588070 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.588036    5002 generic.go:356] "Generic (PLEG): container finished" podID="4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" containerID="bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7" exitCode=0
Oct 30 21:30:08.588227 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.588086    5002 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"
Oct 30 21:30:08.588227 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.588116    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn" event={"ID":"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03","Type":"ContainerDied","Data":"bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7"}
Oct 30 21:30:08.588227 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.588185    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn" event={"ID":"4f5d66fa-39c2-4e5f-b2c7-c1253827ec03","Type":"ContainerDied","Data":"683dc22aa19fe5c967ec324f420f0b2296103db931732e4c47d068d4813d3ee1"}
Oct 30 21:30:08.588227 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.588205    5002 scope.go:122] "RemoveContainer" containerID="bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7"
Oct 30 21:30:08.597993 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.597976    5002 scope.go:122] "RemoveContainer" containerID="bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7"
Oct 30 21:30:08.598265 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:30:08.598247    5002 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7\": container with ID starting with bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7 not found: ID does not exist" containerID="bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7"
Oct 30 21:30:08.598322 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.598273    5002 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7"} err="failed to get container status \"bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7\": rpc error: code = NotFound desc = could not find container \"bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7\": container with ID starting with bd6bc4851ea40c80ac4a06fbe5f19adfabf9c620d7cf42346741b95230f567d7 not found: ID does not exist"
Oct 30 21:30:08.753450 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.753423    5002 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"]
Oct 30 21:30:08.803631 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:08.803602    5002 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0a3a62-7df58f86dc-xw8fn"]
Oct 30 21:30:09.354897 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:09.354874    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:09.627696 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:09.627628    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:09.967315 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:09.967281    5002 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" path="/var/lib/kubelet/pods/4f5d66fa-39c2-4e5f-b2c7-c1253827ec03/volumes"
Oct 30 21:30:10.413792 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:10.413721    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:12.747941 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:12.747907    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:30:16.273380 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:16.273345    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:16.465352 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:16.465319    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:16.730070 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:16.730024    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:22.514791 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:22.514755    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:30:33.046410 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:33.046378    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:30:37.114099 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:30:37.114058    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:30:48.291662 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:48.291588    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:30:53.535197 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:30:53.535161    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:31:07.114230 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:31:07.114192    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:31:24.631828 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:24.631797    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:31:25.444724 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:25.444692    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f74f87f7-vvgwg_467b9a79-f518-4135-af18-337625525bc4/authorino/0.log"
Oct 30 21:31:26.051460 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:26.051420    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bh9rr"]
Oct 30 21:31:37.114144 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:31:37.114107    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:31:37.152803 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:37.152777    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-g97s6_887c992d-ed9b-45a6-9d96-4cec589782db/manager/0.log"
Oct 30 21:31:39.991648 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:39.991620    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d979b6f8b-4bnbl_94038fcc-f568-4af3-adca-9528d7d65ce5/maas-api/0.log"
Oct 30 21:31:42.869358 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:42.869325    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f74f87f7-vvgwg_467b9a79-f518-4135-af18-337625525bc4/authorino/0.log"
Oct 30 21:31:43.343337 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:43.343309    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-w6xjc_983d0d73-eccc-45f1-b2ea-3a12dd3b7860/manager/0.log"
Oct 30 21:31:44.095590 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:44.095565    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8kf5n_f8943322-e55c-454a-a806-55ac57874dbd/manager/0.log"
Oct 30 21:31:44.508766 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:44.508740    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bh9rr_f5998c48-f172-4a4a-ad2c-2687a3e216b9/limitador/0.log"
Oct 30 21:31:44.735804 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:44.735766    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2pc5v_6eb31bb0-64ac-495f-a6b6-353e913ee03f/manager/0.log"
Oct 30 21:31:45.527264 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:45.527239    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp_38948f25-fabf-4bf8-8e26-4520b69f1844/istio-proxy/0.log"
Oct 30 21:31:47.069701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:47.069624    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-b6sv7_ca7680e5-2ded-4caf-a212-2863026c6641/istio-proxy/0.log"
Oct 30 21:31:48.193908 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:48.193878    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-mr976_037d4d37-70b9-4335-9b96-4e5c740a48bc/istio-proxy/0.log"
Oct 30 21:31:48.528263 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:48.528227    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-sf6r9_13afaa6b-b0a8-42ef-8149-33c57717f89e/istio-proxy/0.log"
Oct 30 21:31:48.582327 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:48.582294    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:31:48.754233 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:48.754171    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-tjm7q_c77cbfe3-cbd1-48c7-8685-9745a82600dd/istio-proxy/0.log"
Oct 30 21:31:49.262299 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:49.262239    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl_6b88f899-73f5-4c2f-8af5-9f655fbd6238/istio-proxy/0.log"
Oct 30 21:31:49.934589 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:49.934556    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67cdd9f98c-zf64k_2dfc646e-d42e-44ad-8a5c-09e8115c06ca/router/0.log"
Oct 30 21:31:51.129926 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:51.129901    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5k9gz_8ca301d8-03b2-4873-a5f2-3f3dadf09b75/main/0.log"
Oct 30 21:31:51.669230 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:51.669203    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fkjnq_389982d5-4a66-44c1-bb59-e293269b1e58/main/0.log"
Oct 30 21:31:51.936580 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:51.936510    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6_947c828e-534f-4ec9-b78a-2c354191c54a/main/0.log"
Oct 30 21:31:52.177969 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:52.177945    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-58nn6_7b3937d6-0aa3-4da5-a364-5aff445cf25a/main/0.log"
Oct 30 21:31:52.399503 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:52.399482    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s_8a712478-adcf-48cc-a3ee-eb62d8e5460d/main/0.log"
Oct 30 21:31:53.624973 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:31:53.624945    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f74f87f7-vvgwg_467b9a79-f518-4135-af18-337625525bc4/authorino/0.log"
Oct 30 21:32:03.660077 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:03.660016    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-g97s6_887c992d-ed9b-45a6-9d96-4cec589782db/manager/0.log"
Oct 30 21:32:05.994625 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:05.994578    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d979b6f8b-4bnbl_94038fcc-f568-4af3-adca-9528d7d65ce5/maas-api/0.log"
Oct 30 21:32:07.115154 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:32:07.115122    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:32:08.542792 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:08.542763    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f74f87f7-vvgwg_467b9a79-f518-4135-af18-337625525bc4/authorino/0.log"
Oct 30 21:32:09.015199 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:09.015173    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-w6xjc_983d0d73-eccc-45f1-b2ea-3a12dd3b7860/manager/0.log"
Oct 30 21:32:09.760780 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:09.760757    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8kf5n_f8943322-e55c-454a-a806-55ac57874dbd/manager/0.log"
Oct 30 21:32:10.078138 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:10.078033    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bh9rr_f5998c48-f172-4a4a-ad2c-2687a3e216b9/limitador/0.log"
Oct 30 21:32:10.309262 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:10.309240    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2pc5v_6eb31bb0-64ac-495f-a6b6-353e913ee03f/manager/0.log"
Oct 30 21:32:10.934779 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:10.934755    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp_38948f25-fabf-4bf8-8e26-4520b69f1844/istio-proxy/0.log"
Oct 30 21:32:12.426783 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:12.426666    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-b6sv7_ca7680e5-2ded-4caf-a212-2863026c6641/istio-proxy/0.log"
Oct 30 21:32:13.527322 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:13.527288    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-mr976_037d4d37-70b9-4335-9b96-4e5c740a48bc/istio-proxy/0.log"
Oct 30 21:32:13.842721 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:13.842624    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-sf6r9_13afaa6b-b0a8-42ef-8149-33c57717f89e/istio-proxy/0.log"
Oct 30 21:32:14.070263 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:14.070211    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-tjm7q_c77cbfe3-cbd1-48c7-8685-9745a82600dd/istio-proxy/0.log"
Oct 30 21:32:14.570827 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:14.570800    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl_6b88f899-73f5-4c2f-8af5-9f655fbd6238/istio-proxy/0.log"
Oct 30 21:32:15.244203 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:15.244176    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67cdd9f98c-zf64k_2dfc646e-d42e-44ad-8a5c-09e8115c06ca/router/0.log"
Oct 30 21:32:16.195387 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:16.195355    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5k9gz_8ca301d8-03b2-4873-a5f2-3f3dadf09b75/main/0.log"
Oct 30 21:32:16.703994 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:16.703964    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fkjnq_389982d5-4a66-44c1-bb59-e293269b1e58/main/0.log"
Oct 30 21:32:16.929224 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:16.929197    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clnk6_947c828e-534f-4ec9-b78a-2c354191c54a/main/0.log"
Oct 30 21:32:17.159367 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:17.159340    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-58nn6_7b3937d6-0aa3-4da5-a364-5aff445cf25a/main/0.log"
Oct 30 21:32:17.376209 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:17.376179    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9dl9s_8a712478-adcf-48cc-a3ee-eb62d8e5460d/main/0.log"
Oct 30 21:32:37.115773 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:32:37.115734    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:32:48.929513 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:32:48.929462    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:33:07.114925 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:33:07.114886    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:33:26.767299 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:26.767252    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:33:27.034561 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:27.034429    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:33:36.831189 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:36.831156    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jkspb_96f587e0-8984-434f-b1a5-bae1b56c343b/global-pull-secret-syncer/0.log"
Oct 30 21:33:37.114563 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:33:37.114459    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:33:38.036297 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:38.036269    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-df452_31fce4ed-e652-4919-9cae-157650ca980f/connection-checker/0.log"
Oct 30 21:33:38.467933 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:38.467906    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dqz7k_b85f4408-b319-46f6-aef0-87b5efd91eec/connection-checker/0.log"
Oct 30 21:33:38.895502 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:38.895434    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nr6ns_c2d3fa2f-6503-48f1-a830-375faa8c3a38/connection-checker/0.log"
Oct 30 21:33:39.347314 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:39.347287    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log"
Oct 30 21:33:41.824644 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:41.824593    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-29.us-west-1.compute.internal_a97eddfe748e1e815e76819b9d90360d/haproxy/0.log"
Oct 30 21:33:45.396002 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.395891    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f74f87f7-vvgwg_467b9a79-f518-4135-af18-337625525bc4/authorino/0.log"
Oct 30 21:33:45.715418 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.715391    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-w6xjc_983d0d73-eccc-45f1-b2ea-3a12dd3b7860/manager/0.log"
Oct 30 21:33:45.883283 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883253    5002 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"]
Oct 30 21:33:45.883715 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883701    5002 cpu_manager.go:414] "containerMap: removing container" podUID="0632c00e-df64-4a76-8fa7-dbb9cb09ed00" containerName="payload-processing"
Oct 30 21:33:45.883771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883716    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0632c00e-df64-4a76-8fa7-dbb9cb09ed00" containerName="payload-processing"
Oct 30 21:33:45.883771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883730    5002 cpu_manager.go:414] "containerMap: removing container" podUID="667d8748-f305-4838-a6a2-d6b45bc6d821" containerName="payload-pre-processing"
Oct 30 21:33:45.883771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883735    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667d8748-f305-4838-a6a2-d6b45bc6d821" containerName="payload-pre-processing"
Oct 30 21:33:45.883771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883753    5002 cpu_manager.go:414] "containerMap: removing container" podUID="4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" containerName="maas-api"
Oct 30 21:33:45.883771 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883758    5002 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" containerName="maas-api"
Oct 30 21:33:45.883930 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883845    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="0632c00e-df64-4a76-8fa7-dbb9cb09ed00" containerName="payload-processing"
Oct 30 21:33:45.883930 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883853    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="667d8748-f305-4838-a6a2-d6b45bc6d821" containerName="payload-pre-processing"
Oct 30 21:33:45.883930 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.883859    5002 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5d66fa-39c2-4e5f-b2c7-c1253827ec03" containerName="maas-api"
Oct 30 21:33:45.886968 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.886953    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:45.939177 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.939159    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsvt5\"/\"openshift-service-ca.crt\""
Oct 30 21:33:45.939304 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.939187    5002 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsvt5\"/\"kube-root-ca.crt\""
Oct 30 21:33:45.989849 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.989778    5002 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hsvt5\"/\"default-dockercfg-gxfgx\""
Oct 30 21:33:45.992257 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:45.992235    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"]
Oct 30 21:33:46.014810 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.014784    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5vx\" (UniqueName: \"kubernetes.io/projected/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-kube-api-access-jz5vx\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.014906 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.014857    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-podres\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.014974 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.014914    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-lib-modules\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.014974 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.014945    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-proc\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.015088 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.015035    5002 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-sys\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116348 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116322    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-podres\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116454 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116355    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-lib-modules\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116454 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116377    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-proc\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116454 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116437    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-sys\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116454 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116445    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-podres\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116453    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-proc\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116452    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-lib-modules\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116503    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-sys\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.116647 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.116597    5002 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5vx\" (UniqueName: \"kubernetes.io/projected/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-kube-api-access-jz5vx\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.174202 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.174179    5002 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5vx\" (UniqueName: \"kubernetes.io/projected/1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6-kube-api-access-jz5vx\") pod \"perf-node-gather-daemonset-r5jnm\" (UID: \"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.199057 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.199038    5002 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:46.572183 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.572159    5002 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"]
Oct 30 21:33:46.573390 ip-10-0-141-29 kubenswrapper[5002]: W1030 21:33:46.573362    5002 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a682c94_d4c0_4b1b_8c8c_f73f30a9cda6.slice/crio-78594c0ffaf683f745da4b817664207feec12039a5b2bc10fb028fcd6f8d9a3a WatchSource:0}: Error finding container 78594c0ffaf683f745da4b817664207feec12039a5b2bc10fb028fcd6f8d9a3a: Status 404 returned error can't find the container with id 78594c0ffaf683f745da4b817664207feec12039a5b2bc10fb028fcd6f8d9a3a
Oct 30 21:33:46.974659 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.974623    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm" event={"ID":"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6","Type":"ContainerStarted","Data":"8cc5e3ff4fda69083b26a67a310e0354e0078fdea976024bd4117e3c70beeaf2"}
Oct 30 21:33:46.974813 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.974668    5002 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm" event={"ID":"1a682c94-d4c0-4b1b-8c8c-f73f30a9cda6","Type":"ContainerStarted","Data":"78594c0ffaf683f745da4b817664207feec12039a5b2bc10fb028fcd6f8d9a3a"}
Oct 30 21:33:46.974813 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:46.974778    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:47.003720 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:47.003699    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8kf5n_f8943322-e55c-454a-a806-55ac57874dbd/manager/0.log"
Oct 30 21:33:47.093701 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:47.093664    5002 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm" podStartSLOduration=2.093652686 podStartE2EDuration="2.093652686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:47.09241109 +0000 UTC m=+4453.241462078" watchObservedRunningTime="2026-10-30 21:33:47.093652686 +0000 UTC m=+4453.242703674"
Oct 30 21:33:47.403260 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:47.403191    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bh9rr_f5998c48-f172-4a4a-ad2c-2687a3e216b9/limitador/0.log"
Oct 30 21:33:47.865523 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:47.865499    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2pc5v_6eb31bb0-64ac-495f-a6b6-353e913ee03f/manager/0.log"
Oct 30 21:33:49.367552 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:49.367502    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:33:52.990794 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:52.990761    5002 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-r5jnm"
Oct 30 21:33:53.414362 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:53.414289    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-9v6j5_0bb8ae2c-9126-4e64-b930-c748b4a44bf1/cluster-monitoring-operator/0.log"
Oct 30 21:33:56.265089 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:56.265008    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:33:56.443274 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:56.443246    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:33:57.321595 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:57.321572    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hg6rr_72869d66-7380-4336-92f3-fd98774219de/node-exporter/0.log"
Oct 30 21:33:57.738073 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:57.738047    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hg6rr_72869d66-7380-4336-92f3-fd98774219de/kube-rbac-proxy/0.log"
Oct 30 21:33:58.155042 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:33:58.154970    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hg6rr_72869d66-7380-4336-92f3-fd98774219de/init-textfile/0.log"
Oct 30 21:34:07.114670 ip-10-0-141-29 kubenswrapper[5002]: E1030 21:34:07.114632    5002 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cmnm9" containerName="extractor"
Oct 30 21:34:08.081686 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:34:08.081656    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:34:08.265982 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:34:08.265945    5002 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l4ldp_c23bf1a3-ba4f-40cb-b8e0-c9df7e43bd0f/konnectivity-agent/0.log" limit=1024
Oct 30 21:34:08.759353 ip-10-0-141-29 kubenswrapper[5002]: I1030 21:34:08.759322    5002 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp_38948f25-fabf-4bf8-8e26-4520b69f1844/istio-proxy/0.log"